Simon Willison (Django)Fast Autocomplete Search for Your Website

Fast Autocomplete Search for Your Website

I wrote a tutorial for the 24 ways advent calendar on building fast autocomplete search for a website on top of Datasette and SQLite. I built the demo against 24 ways itself - I used wget to recursively fetch all 330 articles as HTML, then wrote code in a Jupyter notebook to extract the raw data from them (with BeautifulSoup) and load them into SQLite using my sqlite-utils Python library. I deployed the resulting database using Datasette, then wrote some vanilla JavaScript to implement autocomplete using fast SQL queries against the Datasette JSON API.

Via @simonw

Amazon Web ServicesServerless and startups, the beginning of a beautiful friendship

Guest post by AWS Serverless Hero Slobodan Stojanović. Slobodan is the co-author of the book Serverless Applications with Node.js; CTO of Cloud Horizon, a software development studio; and CTO of Vacation Tracker, a Slack-based, leave management app. Slobodan is excited with serverless because it allows him to build software faster and cheaper. He often writes about serverless and talks about it at conferences.

Serverless seems to be perfect for startups. The pay-per-use pricing model and infrastructure that costs you nothing if no one is using your app makes it cheap for early-stage startups.

On the other side, it’s fully managed and scales automatically, so you don’t have to be afraid of large marketing campaigns or unexpected traffic. That’s why we decided to use serverless when we started working on our first product: Vacation Tracker.

Vacation Tracker is a Slack-based app that helps you to track and manage your team’s vacations and days off. Both our Slack app and web-based dashboard needed an API, so an AWS Lambda function with an Amazon API Gateway trigger was a logical starting point. API Gateway provides a public API. Each time that the API receives the request, the Lambda function is triggered to answer that request.

Our app is focused on small and medium teams and is the app that you use less than a few times per day. Periodic usage makes the pay-per-use, serverless pricing model a big win for us because both API Gateway and Lambda cost $0 initially.

Start small, grow tall

We decided to start small, with a simple prototype. Our prototype was a real Slack app with a few hardcoded actions and a little calendar. We used Claudia.js and Bot Builder to build it. Claudia.js is a simple tool for the deployment of Node.js serverless functions to Lambda and API Gateway.

After we finished our prototype, we published a landing page. But we continued building our product even as users signed up for the closed beta access.

Just a few months later, we had a bunch of serverless functions in production: chatbot, dashboard API, Slack notifications, a few tasks for Stripe billing, etc. Each of these functions had their own triggers and roles. While our app was working without issues, it was harder and harder to deploy a new stage.

It was clear that we had to organize our app better. So, our Vacation Tracker koala met the AWS SAM squirrel.

Herding Lambda functions

We started by mapping all our services. Then, we tried to group them into flows. We decided to migrate piece by piece to AWS Serverless Application Model (AWS SAM), an open-source framework for building serverless apps on AWS. As AWS SAM is language-agnostic, it still doesn’t know how to handle Node.js dependencies. We used Claudia’s pack command as a build step.

Grouping services in serverless apps brought back easier deployments. With just a single command, we had a new environment ready for our tester.

Soon after, we had AWS CloudFormation templates for a Slack chatbot, an API, a Stripe-billing based payment flow, notifications flow, and a few other flows for our app.

Like other startups, Vacation Tracker’s goal is to be able to evolve fast and adapt to user needs. To do so, you often need to run experiments and change things on the fly. With that in mind, our goal was to extract some common functionalities from the app flow to reusable components.

For example, as we used multiple Slack slash commands in some of the experiments, we extracted it from the Slack chatbot flow into a reusable component.

Our Slack slash command component consists of a few elements:

  • An API Gateway API with routes for the Slack slash command and message action webhooks
  • A Lambda function that handles slash commands
  • A Lambda function that handles message actions
  • An Amazon SNS topic for parsed Slack data

With this component, we can run our slash command experiments faster. Adding a new Slack slash command to our app requires the deployment of the slash command. We also wrote a few Lambda functions that are triggered by the SNS topic or handle business logic.

While working on Vacation Tracker, we realized the potential value of reusable components. Imagine how fast you would be able to assemble your MVP if you could use the standard components that someone else built? Building an app would require writing glue between reused parts and focus on a business logic that makes your app unique.

This dream can become a reality with AWS Serverless Application Repository, a repository for open source serverless components.

Instead of dreaming, we decided to publish a few reusable components to the Serverless Application Repository, starting with the Slack slash command app. But to do so, we had to have a well-tested app, which led us to the next challenge: how to architect and test a reusable serverless app?

Hexagonal architecture to the rescue

Our answer was simple: Hexagonal architecture, or ports and adapters. It is a pattern that allows an app to be equally driven by users, programs, automated tests, or batch scripts. The app can be developed and tested in isolation from its eventual runtime devices and databases. This makes hexagonal architecture a perfect fit for microservices and serverless apps.

Applying this to Vacation Tracker, we ended up with a setup similar to the following diagram. It consists of the following:

  • lambda.js and main.js files. lambda.js has no tests, as it simply wires the dependencies, such as sns-notification-repository.js, and invokes main.js.
  • main.js has its own unit and integration tests. Integration tests are using local integrations.
  • Each repository has its own unit and integration tests. In their integration tests, repositories connect to AWS services. For example, sns-notification-repository.js integration tests connect to Amazon SNS.

Each of our functions has at least two files: lambda.js and main.js. The first file is small and just invokes main.js with all the dependencies (adapters). This file doesn’t have automated tests, and it looks similar to the following code snippet:

const {
 httpResponse,
 SnsNotificationRepository
} = require('@serverless-slack-command/common')
const main = require('./main')
async function handler(event) {
  const notification = new SnsNotificationRepository(process.env.notificationTopic)
  await main(event.body, event.headers, event.requestContext, notification)
  return httpResponse()
}

exports.handler = handler

The second, and more critical file of each function is main.js. This file contains the function’s business logic, and it must be well tested. In our case, this file has its own unit and integration tests. But the business logic often relies on external integrations, for example sending an SNS notification. Instead of testing all external notifications, we test this file with other adapters, such as a local notification repository.

This file looks similar to the following code snippet:

const qs = require('querystring')

async function slashCommand(slackEvent, headers, requestContext, notification) {
  const eventData = qs.parse(slackEvent);
  return await notification.send({
    type: 'SLASH_COMMAND',
    payload: eventData,
    metadata: {
      headers,
      requestContext
    }
  })
}

module.exports = slashCommand

Adapters for external integrations have their own unit and integration tests, including tests that check the integration with the AWS service. This way we minimized the number of tests that rely on AWS services but still kept our app covered with all necessary tests.

And they lived happily ever after…

Migration to AWS SAM simplified and improved our deployment process. Setting up a new environment now takes minutes, and it can be additionally reduced in future by nesting AWS CloudFormation stacks. Development and testing for our components are easy using hexagonal architecture. Reusable components and Serverless Application Repository put the cherry on top of our serverless cake.

This could be the start of a beautiful friendship between serverless and startups. With serverless, your startup infrastructure is fully managed, and you pay it only if someone is using your app. The serverless pricing model allows you to start cheap. With Serverless Application Repository, you can build your MVPs faster, as you can reuse existing components. These combined benefits give you superpowers and enough velocity to be able to compete with other products with larger teams and budgets.

We are happy to see what startups can build (and outsource) using Serverless Application Repository.

In the meantime, you can see the source of our first open source serverless component on GitHub: https://github.com/vacationtracker/serverless-slack-slash-command-app.

And if you want to try Vacation Tracker, visit https://vacationtracker.io, and you can double your free trial period using the AWS_IS_AWESOME promo code.

Simon Willison (Django)Develop Your Naturalist Superpowers with Observable Notebooks and iNaturalist

Develop Your Naturalist Superpowers with Observable Notebooks and iNaturalist

Natalie's article for this year's 24 ways advent calendar shows how you can use Observable notebooks to quickly build interactive visualizations against web APIs. She uses the iNaturalist API to show species of Nudibranchs that you might see in a given month, plus a Vega-powered graph of sightings over the course of the year. This really inspired me to think harder about how I can use Observable to solve some of my API debugging needs, and I've already spun up a couple of private Notebooks to exercise new APIs that I'm building at work. It's a huge productivity boost.

Via @natbat

ProgrammableWebFitbit OS 3.0 Includes Four New Developer APIs

Fitbit has released Fitbit OS 3.0 for developers. The 3.0 SDK includes a suite of APIs that allow developers to create for the Fitbit Ionic and Versa smartwatches. The APIs allow applications to computer on-device calculations, launch third party applications, upload files, enable side-loading, and more.

3.0 specifically includes 4 new APIs:

ProgrammableWebBox Skills Kit Moves to General Availability

Box has just announced the general availability of Box Skills Kit, a framework for artificial intelligence integration that provides developers with powerful new ways to use AI to analyze and enhance users content on Box. The company also announced new partnerships that aim to simplify the design and implementation of custom Box Skills solutions. 

ProgrammableWebArrivedo API Empowers Hotels with Localized Experience and Content Materials

Arrivedo recently announced an API that gives hotels the ability to curate and suggest local content and experiences for guests. Hotels can pull from Arrivedo's content of stories, routes, maps, places, photos and videos to create hotel-specific packages, guides, etc. Hotels can create and share their Arrivedo-powered marketing materials through websites, emails, in-room smart devices, mobile apps, training materials, and more.

Amazon Web ServicesBoost your infrastructure with the AWS CDK

This guest post is by AWS Container Hero Philipp Garbe. Philipp works as Lead Platform Engineer at Scout24 in Germany. He is driven by technologies and tools that allow him to release faster and more often. He expects that every commit automatically goes into production. You can find him on Twitter at @pgarbe.

Infrastructure as code (IaC) has been adopted by many teams in the last few years. It makes provisioning of your infrastructure easy and helps to keep your environments consistent.

But by using declarative templates, you might still miss many practices that you are used to for “normal” code. You’ve probably already felt the pain that each AWS CloudFormation template is just a copy and paste of your last projects or from StackOverflow. But can you trust these snippets? How can you align improvements or even security fixes through your code base? How can you share best practices within your company or the community?

Fortunately for everyone, AWS published the beta for an important addition to AWS CloudFormation: the AWS Cloud Development Kit (AWS CDK).

What’s the big deal about the AWS CDK?

All your best practices about how to write good AWS CloudFormation templates can now easily be shared within your company or the developer community. At the same time, you can also benefit from others doing the same thing.

For example, think about Amazon DynamoDB. Should be easy to set up in AWS CloudFormation, right? Just some lines in your template. But wait. When you’re already in production, you realize that you’ve got to set up automatic scaling, regular backups, and most importantly, alarms for all relevant metrics. This can amount to several hundred lines.

Think ahead: Maybe you’ve got to create another application that also needs a DynamoDB database. Do you copy and paste all that YAML code? What happens later, when you find some bugs in your template? Do you apply the fix to both code bases?

With the AWS CDK, you’re able to write a “construct” for your best practice, production-ready DynamoDB database. Share it as an npm package with your company or anyone!

What is the AWS CDK?

Back up a step and see what the AWS CDK looks like. Compared to the declarative approach with YAML (or JSON), the CDK allows you to declare your infrastructure imperatively. The main language is TypeScript, but several other languages are also supported.

This is what the Hello World example from Hello, AWS CDK! looks like:

import cdk = require('@aws-cdk/cdk');
import s3 = require('@aws-cdk/aws-s3');

class MyStack extends cdk.Stack {
    constructor(parent: cdk.App, id: string, props?: cdk.StackProps) {
        super(parent, id, props);

        new s3.Bucket(this, 'MyFirstBucket', {
            versioned: true
        });
    }
}

class MyApp extends cdk.App {
    constructor(argv: string[]) {
        super(argv);

        new MyStack(this, 'hello-cdk');
    }
}

new MyApp().run();

Apps are the root constructs and can be used directly by the CDK CLI to render and deploy the AWS CloudFormation template.

Apps consist of one or more stacks that are deployable units and contains information about the Region and account. It’s possible to have an app that deploys different stacks to multiple Regions at the same time.

Stacks include constructs that are representations of AWS resources like a DynamoDB table or AWS Lambda function.

A lib is a construct that typically encapsulates further constructs. With that, higher class constructs can be built and also reused. As the construct is just TypeScript (or any other supported language), a package can be built and shared by any package manager.

Constructs

As the CDK is all about constructs, it’s important to understand them. It’s a hierarchical structure called a construct tree. You can think of constructs in three levels:

Level 1: AWS CloudFormation resources

This is a one-to-one mapping of existing resources and is automatically generated. It’s the same as the resources that you use currently in YAML. Ideally, you don’t have to deal with these constructs directly.

Level 2: The AWS Construct Library

These constructs are on an AWS service level. They’re opinionated, well-architected, and handwritten by AWS. They come with proper defaults and should make it easy to create AWS resources without worrying too much about the details.

As an example, this is how to create a complete VPC with private and public subnets in all available Availability Zones:

import ec2 = require('@aws-cdk/aws-ec2'); 

const vpc = new ec2.VpcNetwork(this, 'VPC');

The AWS Construct Library has some nice concepts about least privilege IAM policies, event-driven API actions, security groups, and metrics. For example, IAM policies are automatically created based on your intent. When a Lambda function subscribes to an SNS topic, a policy is created that allows the topic to invoke the function.

AWS services that offer Amazon CloudWatch metrics have functions like metricXxx() and return metric objects that can easily be used to create alarms.

new Alarm(this, 'Alarm', {
    metric: fn.metricErrors(),
    threshold: 100,
    evaluationPeriods: 2,
});

For more information, see AWS Construct Library.

Level 3: Your awesome stuff

Here’s where it gets interesting. As mentioned earlier, constructs are hierarchical. They can be higher-level abstractions based on other constructs. For example, on this level, you can write your own Amazon ECS cluster construct that contains automatic node draining, automatic scaling, and all the right alarms. Or you can write a construct for all necessary alarms that an Amazon RDS database should monitor. It’s up to you to create and share your constructs.

Conclusion

It’s good that AWS went public in an early stage. The docs are already good, but not everything is covered yet. Not all AWS services have an AWS Construct Library module defined (level 2). Many have only the pure AWS CloudFormation constructs (level 1).

Personally, I think the AWS CDK is a huge step forward, as it allows you to re-use AWS CloudFormation code and share it with others. It makes it easy to apply company standards and allows people to work on awesome features and spend less time on writing “boring” code.

Amazon Web ServicesPick the Right Tool for your IT Challenge

This guest post is by AWS Community Hero Markus Ostertag. As CEO of the Munich-based ad-tech company Team Internet AG, Markus is always trying to find the best ways to leverage the cloud, loves to work with cutting-edge technologies, and is a frequent speaker at AWS events and the AWS user group Munich that he co-founded in 2014.

Picking the right tools or services for a job is a huge challenge in IT—every day and in every kind of business. With this post, I want to share some strategies and examples that we at Team Internet used to leverage the huge “tool box” of AWS to build better solutions and solve problems more efficiently.

Use existing resources or build something new? A hard decision

The usual day-to-day work of an IT engineer, architect, or developer is building a solution for a problem or transferring a business process into software. To achieve this, we usually tend to use already existing architectures or resources and build an “add-on” to it.

With the rise of microservices, we all learned that modularization and decoupling are important for being scalable and extendable. This brought us to a different type of software architecture. In reality, we still tend to use already existing resources, like the same database of existing (maybe not fully used) Amazon EC2 instances, because it seems easier than building up new stuff.

Stacks as “next level microservices”?

We at Team Internet are not using the vocabulary of microservices but tend to speak about stacks and building blocks for the different use cases. Our approach is matching the idea of microservices to everything, including the database and other resources that are necessary for the specific problem we need to address.

It’s not about “just” dividing the software and code into different modules. The whole infrastructure is separated based on different needs. Each of those parts of the full architecture is our stack, which is as independent as possible from everything else in the whole system. It only communicates loosely with the other stacks or parts of the infrastructure.

Benefits of this mindset = independence and flexibility

  • Choosing the right parts. For every use case, we can choose the components or services that are best suited for the specific challenges and don’t need to work around limitations. This is especially true for databases, as we can choose from the whole palette instead of trying to squeeze requirements into a DBMS that isn’t built for that. We can differentiate the different needs of workloads like write-heavy vs. read-heavy or structured vs. unstructured data.
  • Rebuilding at will. We’re flexible in rebuilding whole stacks as they’re only loosely coupled. Because of this, a team can build a proof-of-concept with new ideas or services and run them in parallel on production workload without interfering or harming the production system.
  • Lowering costs. Because the operational overhead of running multiple resources is done by AWS (“No undifferentiated heavy lifting”), we just need to look at the service pricing. Most of the price schemes at AWS are supporting the stacks. For databases, you either pay for throughput (Amazon DynamoDB) or per instance (Amazon RDS, etc.). On the throughput level, it’s simple as you just split the throughput you did on one table to several tables without any overhead. On the instance level, the pricing is linear so that an r4.xlarge is half the price of an r4.2xlarge. So why not run two r4.xlarge and split the workload?
  • Designing for resilience. This approach also helps your architecture to be more reliable and resilient by default. As the different stacks are independent from each other, the scaling is much more granular. Scaling on larger systems is often provided with a higher “security buffer,” and failures (hardware, software, fat fingers, etc.) only happen on a small part of the whole system.
  • Taking ownership. A nice side effect we’re seeing now as we use this methodology is the positive effect on ownership and responsibility for our teams. Because of those stacks, it is easier to pinpoint and fix issues but also to be transparent and clear on who is responsible for which stack.

Benefits demand efforts, even with the right tool for the job

Every approach has its downsides. Here, it is obviously the additional development and architecture effort that needs to be taken to build such systems.

Therefore, we decided to always have the goal of a perfect system with independent stacks and reliable and loosely coupled processes between them in our mind. In reality, we sometimes break our own rules and cheat here and there. Even if we do, to have this approach helps us to build better systems and at least know exactly at what point we take a risk of losing the benefits. I hope the explanation and insights here help you to pick the right tool for the job.

ProgrammableWebIntroduction to the ipapi Geolocation API

The Importance of IP Geolocation for Web Developers

Today, I'm introducing ipapi, a new service that provides a wealth of information about your website's visitors based on their IP address. Using the service, you can determine the customer's location, timezone, default language and currency and details about their network's organizational identity. And, ipapi servers provide this information with extremely fast response times.

Simon Willison (Django)PEP 8016 -- The Steering Council Model

PEP 8016 -- The Steering Council Model

The votes are in and Python has a new governance model, partly inspired by the model used by the Django Software Foundation. A core elected council of five people (with a maximum of two employees from any individual company) will oversee the project.

Via Python governance vote (December 2018): Results - Committers - Discussions on Python.org

Simon Willison (Django)Pampy: Pattern Matching for Python

Pampy: Pattern Matching for Python

Ingenious implementation of Erlang/Rust style pattern matching in just 150 lines of extremely cleanly designed and well-tested Python.

Via Hacker News

ProgrammableWebCloudflare Announces Mobile SDK Supports Mobile App Traffic Acceleration

Cloudflare, an internet performance and security provider, announced that early access to Traffic Acceleration was being added to its Cloudflare Mobile SDK. The traffic acceleration technology aims to address network throughput for mobile apps thereby increasing performance beyond what would be seen with TCP.

Jeremy Keith (Adactio)Browsers

Microsoft’s Edge browser is going to switch its rendering engine over to Chromium.

I am deflated and disappointed.

There’s just no sugar-coating this. I’m sure the decision makes sound business sense for Microsoft, but it’s not good for the health of the web.

Very soon, the vast majority of browsers will have an engine that’s either Blink or its cousin, WebKit. That may seem like good news for developers when it comes to testing, but trust me, it’s a sucky situation of innovation and agreement. Instead of a diverse browser ecosystem, we’re going to end up with incest and inbreeding.

There’s one shining exception though. Firefox. That browser was originally created to combat the seemingly unstoppable monopolistic power of Internet Explorer. Now that Microsoft are no longer in the rendering engine game, Firefox is once again the only thing standing in the way of a complete monopoly.

I’ve been using Firefox as my main browser for a while now, and I can heartily recommend it. You should try it (and maybe talk to your relatives about it at Christmas). At this point, which browser you use no longer feels like it’s just about personal choice—it feels part of something bigger; it’s about the shape of the web we want.

Jeffrey wrote that browser diversity starts with us:

The health of Firefox is critical now that Chromium will be the web’s de facto rendering engine.

Even if you love Chrome, adore Gmail, and live in Google Docs or Analytics, no single company, let alone a user-tracking advertising giant, should control the internet.

Andy Bell also writes about browser diversity:

I’ll say it bluntly: we must support Firefox. We can’t, as a community allow this browser engine monopoly. We must use Firefox as our main dev browsers; we must encourage our friends and families to use it, too.

Yes, it’s not perfect, nor are Mozilla, but we can help them to develop and grow by using Firefox and reporting issues that we find. If we just use and build for Chromium, which is looking likely (cough Internet Explorer monopoly cough), then Firefox will fall away and we will then have just one major engine left. I don’t ever want to see that.

Uncle Dave says:

If the idea of a Google-driven Web is of concern to you, then I’d encourage you to use Firefox. And don’t be a passive consumer; blog, tweet, and speak about its killer features. I’ll start: Firefox’s CSS Grid, Flexbox, and Variable Font tools are the best in the business.

Mozilla themselves came out all guns blazing when they said Goodbye, EdgeHTML:

Microsoft is officially giving up on an independent shared platform for the internet. By adopting Chromium, Microsoft hands over control of even more of online life to Google.

Tim describes the situation as risking a homogeneous web:

I don’t think Microsoft using Chromium is the end of the world, but it is another step down a slippery slope. It’s one more way of bolstering the influence Google currently has on the web.

We need Google to keep pushing the web forward. But it’s critical that we have other voices, with different viewpoints, to maintain some sense of balance. Monocultures don’t benefit anyone.

Andre Alves Garzia writes that while we Blink, we lose the web:

Losing engines is like losing languages. People may wish that everyone spoke the same language, they may claim it leads to easier understanding, but what people fail to consider is that this leads to losing all the culture and way of thought that that language produced. If you are a Web developer smiling and happy that Microsoft might be adopting Chrome, and this will make your work easier because it will be one less browser to test, don’t be! You’re trading convenience for diversity.

I like that analogy with language death. If you prefer biological analogies, it’s worth revisiting this fantastic post by Rachel back in August—before any of us knew about Microsoft’s decision—all about the ecological impact of browser diversity:

Let me be clear: an Internet that runs only on Chrome’s engine, Blink, and its offspring, is not the paradise we like to imagine it to be.

That post is a great history lesson, documenting how things can change, and how decisions can have far-reaching unintended consequences.

So these are the three browser engines we have: WebKit/Blink, Gecko, and EdgeHTML. We are unlikely to get any brand new bloodlines in the foreseeable future. This is it.

If we lose one of those browser engines, we lose its lineage, every permutation of that engine that would follow, and the unique takes on the Web it could allow for.

And it’s not likely to be replaced.

Henry SivonenRust 2019

The Rust team encouraged people to write blog posts reflecting on Rust in 2018 and proposing goals and directions for 2019. Here’s mine.

This is knowingly blatantly focused on the niche that is immediately relevant to my work. I don’t even pretend this to represent any kind of overall big picture.

Rust in 2018

In my Rust 2018 post, I had these items:

  • simd-Style SIMD
  • Rust bool in FFI is C _Bool
  • Debug Info for Code Expanded from Macros
  • Non-Nightly Benchmarking
  • GUI for rr replay
  • Tool for Understanding What LLVM Did with a Given Function

As far as I know, the kind of tool I wanted for understading what LLVM did does not exist in a way that does not involve extracting a minimized case with the dependencies for copying and pasting to rust.godbolt.org. After one goes through the effort of making a Compiler Explorer-compatible extract, the tool is great, though. I don’t know if the feature existed a year ago, but Compiler Explorer now has tooltips that explain what assembly instructions do, so I’d rate this old wish half fulfilled. (Got the asm explanations but didn’t get to avoid manually extracting the code under scrutiny.)

I’ve been told that GUIs for rr exist and work. However, I got stuck with cgdb (launch with rr replay --debugger=/usr/bin/cgdb --no-redirect-output; thanks to Thomas McGuire and David Faure of KDAB for that incantation), because it has worked well for me, and the Python+browser front end that was recommended to me did not work right away. (I should try again.)

Also, Rust bool is now documented to have size_of 1 and the proposal to make the compiler complain about bool in FFI has been abandoned. 🎉

Cool Things in 2018 That I Did Not Ask For

Looking back at 2018 beyond what I wrote in my Rust 2018 post, I am particularly happy about these features making it to non-nightly Rust:

Non-lexical lifetimes is a huge boost for the ergonomics of the language. I hope the people who previously turned away from Rust due to the borrow checker will be willing to try again.

align_to makes it easier to write more obviously correct optimizations that look at byte buffers one register at a time. A bit disappointly, the previous sentence cannot say “safe code”, because align_to is still unsafe. It would be nice if there was a safe version with a trait bound on the types requiring types whose all bit patterns are defined and then having primitive integers and SIMD vectors with primitive integer lane types implement the relevant marker trait. (I.e. exposing endianness would be considered safe like integer overflow is considered safe.)

I expect chunks_exact to be relevant to writing safe SIMD code.

Carry-Overs from 2018

Some items from a year ago are not done.

Non-Nightly Benchmarking

The library support for the cargo bench feature has been in the state “basically, the design is problematic, but we haven’t had anyone work through those issues yet” since 2015. It’s a useful feature nonetheless. Like I said a year ago, it’s time to let go of the possibility of tweaking it for elegance and just let users use it on non-nighly Rust.

Debug Info for Code Expanded from Macros

No news on this RFC.

Portable SIMD

A lot of work has been done on this topic in the past year, which is great. Thank you! Instead of the design of the simd crate, the design and implementation is proceeding in the packed_simd crate. I wish that packed_simd with its into_bits feature enabled becomes code::simd / std::simd and available on non-nightly Rust in 2019.

A year ago I wished that core::arch / std::arch did not become available on non-nightly Rust before core::simd / std::simd out of concern that vendor-specific SIMD shipping before portable SIMD would unnecessarily skew the ecosystem towards the incumbent (Intel). I think it is too early to assess if the concern was valid.

New Items

In addition to reiterating the old items, I do have some new ones, too.

Compiling the Standard Library with User Settings

At present, when you compile a Rust artifact, your own code and the crates your code depends on get compiled, but the standard library is taken as a pre-compiled library. This is problematic especially with SIMD functionality moving to the standard library.

32-bit CPU architectures like x86, ARM, PowerPC and MIPS introduced SIMD during the evolution of the instruction set architecture. Therefore, unlike in the case of x86_64, aarch64 and little-endian POWER, generic 32-bit targets cannot assume that SIMD support is present. If you as an application developer decide to scope your application to support only recent enough 32-bit CPUs that you can assume SSE2/NEON/AltiVec/MSA to be present and want to use packed_simd / std::simd to use the SIMD capability of the CPU, you are going to have a bad time if the Rust standard library has been compiled with the assumption that the SIMD unit does not exist.

For 32-bit x86 and SSE2 Rust solves this by providing two targets: i586 without SSE2 and i686 with SSE2. Currently, the ARMv7 (both Thumb2 and non-Thumb2) targets are without NEON. I am hoping to introduce Thumb2+NEON variants in 2019.

Adding targets won’t scale, though. For example, even in the x86_64 case you might determine that it is OK for you application to require a CPU that supports SSSE3, which is relevant to portable SIMD by providing arbitrary shuffling as a single instruction. (At present, the SSE2 shuffle generation back end for LLVM misses even some seemingly obvious cases like transposing each of the eight pairs of lanes in u8x16 by lane-wise shifting by 8 to both directions in an u16x8 interpretation and bitwise ORing the results.)

I hope that in 2019, Cargo gains the Xargo functionality of being able to compile the standard library with the same target feature settings that are used for compiling the user code and the crate dependencies.

Better Integer Range Analysis for Bound Check Elision

Currently, LLVM only elides the bound checks when indexing into slices if you’ve made the most obvious comparison previously between the index and the slice length. For example:

if i < slice.len() {
    slice[i] // bound check elided
}

Pretty much anything more complex results in a bound check branch, and the performance effect is measurable when it happens in the innermost loop. I hope that rustc and LLVM will do better in 2019. Specifically:

  • LLVM should become able to eliminate the second check in code like:

    if a + C < b {
        if a + D < b {
        	// ...
        }
    }

    …if a, b, C, and D are all of type usize, a and b are run-time variables, C and D are compile-time constants such that D <= C and a + C can be proven at compile time not to overflow.

  • LLVM should become able to figure out that a + C didn’t overflow if it was written as a.checked_add(C).unwrap() and execution continued to the second check.

  • rustc should become able to tell LLVM that a small constant added to slice.len() or a value previously checked to be less than slice.len() does not overflow by telling LLVM to assume that the maximum possible value for a slice length is quite a bit less than usize::max_value().

    Since a slice has to represent a possible allocation, the maximum possible value for len() is not usize::max_value(). On 64-bit platforms, rustc should tell LLVM that the usize returned by len() is capped by the number of bits the architecture actually uses for the virtual address space, which is lower than 64 bits. I’m not sure if Rust considers it permissible for 32-bit PAE processes allocate more than half the address space in a single allocation (it seems like a bad thing to allow in terms of pointer difference computations, but it looks like glibc has at least it the past allowed such allocations), but even if it considered permissible, it should be possible to come up with a slice size limit by observing that a slice cannot fill the whole address space, because at least the stack size and the size of the code for a minimal program have to be reserved.

  • LLVM should become able to figure out that if a: ufoo and a >= C, then a - C < ufoo::max_size() + 1 - C and, therefore, indexing with a - C into an array whose length is ufoo::max_size() + 1 - C does not need a bound check. (Where C is a compile-time constant.)

likely() and unlikely() for Plain if Branch Prediction Hints

The issue for likely() and unlikely() has stalled on the observation that they don’t generalize for if let, match, etc. They would work for plain if, though. Let’s have them for plain if in 2019 even if if let, match, etc., remain unaddressed for now.

No LTS

Rust has successfully delivered on “stability without stagnation” to the point that Red Hat has announced Rust updates for RHEL on a 3-month frequency instead of Rust getting stuck for the duration of the lifecycle of a RHEL version. That is, contrary to popular belief, the “stability” part works without an LTS. At this point, doing an LTS would be a stategic blunder that would jeopardize the “without stagnation” part.

ProgrammableWebDaily API RoundUp: Geospark, Mirrorfly, ChainFront, CloudPublish

Every day, the ProgrammableWeb team is busy, updating its three primary directories for APIs, clients (language-specific libraries or SDKs for consuming or providing APIs), and source code samples.

Simon Willison (Django)Quoting Bryan Cantrill

for those open source companies that still harbor magical beliefs, let me put this to you as directly as possible: cloud services providers are emphatically not going to license your proprietary software. I mean, you knew that, right? The whole premise with your proprietary license is that you are finding that there is no way to compete with the operational dominance of the cloud services providers; did you really believe that those same dominant cloud services providers can’t simply reimplement your LDAP integration or whatever? The cloud services providers are currently reproprietarizing all of computing — they are making their own CPUs for crying out loud! — reimplementing the bits of your software that they need in the name of the service that their customers want (and will pay for!) won’t even move the needle in terms of their effort.

Bryan Cantrill

ProgrammableWebFacebook Photos API Bug Exposed Private Images of 6.8M Users

In yet another blunder, Facebook announced today that a bug in one of their APIs allowed third-party apps to access the private images of up to 6.8 million users. From September 13 to September 25, over 1,500 apps had access to photos that users never authorized.

ProgrammableWebArm and Disarm Home Security Systems with Alexa

Amazon recently announced its Alexa Security Panel Controller API. The API allows users to connect security systems to Alexa. Instead of manually punching in your pin and then pressing the "arm" button, imagine turning on your security system by saying, "Alexa, arm my security system."

ProgrammableWebChainFront Launches New Blockchain APIs and Developer Portal

ChainFront, a blockchain API-as-a-service provider, has announced the release of new public APIs and a developer portal. With the launch of these new ChainFront APIs, the company is helping developers build applications that require blockchain integration by handling the tasks of backing up keys and keeping track of on-device wallets.

ProgrammableWeb: APIsBDApps Pro

BDApps is an application development platform that provides Robi network tools for monetization and messaging. BDApps Pro supports endpoints for SMS, USSD sessions, subscriptions, and micro-payments. The API requires JSON objects for requests and responses. BDApps is based in Bangladesh.
Date Updated: 2018-12-14
Tags: Application Development, Messaging, , Payments, , Subscriptions

ProgrammableWebBuild Group Messaging Apps with the First and Only Group Messaging API

Note: The demo here assumes at least a basic knowledge of Node.js and Express.

Introduction

ProgrammableWebDaily API RoundUp: Nomics, ClimBiz, Chattermill, SignEasy, XERO

Every day, the ProgrammableWeb team is busy, updating its three primary directories for APIs, clients (language-specific libraries or SDKs for consuming or providing APIs), and source code samples.

ProgrammableWebKong Launches Managed Cloud Service

Kong, the API platform formerly known as Mashape, recently announced the launch of Kong Cloud. This new service targets organizations that want the API Management capabilities of Kong Enterprise, but, akin to the benefits of other SaaS apps, don't want the burden of managing the software itself (new version updates, bug fixes, capacity management, etc.)

Henry Sivonenencoding_rs

A Web-Compatible Character Encoding Library in Rust. (Used in Firefox.)

Amazon Web ServicesUsing AWS AI and Amazon Sumerian in IT Education

This guest post is by AWS Machine Learning Hero, Cyrus Wong. Cyrus is a Data Scientist at the Hong Kong Institute of Vocational Education (Lee Wai Lee) Cloud Innovation Centre. He has achieved all nine AWS Certifications and enjoys sharing his AWS knowledge with others through open-source projects, blog posts, and events.

Our institution (IVE) provides IT training to several thousand students every year and one of our courses successfully applied AWS Promotional Credits. We recently built an open-source project called “Lab Monitor,” which uses AWS AI, serverless, and AR/VR services to enhance our learning experience and gather data to understand what students are doing during labs.

Problem

One of the common problems of lab activity is that students are often doing things that have nothing to do with the course (such as watching videos or playing games). And students can easily copy answers from their classmate because the lab answers are in softcopy. Teachers struggle to challenge students as there is only one answer in general. No one knows which students are working on the lab or which are copying from one another!

Solution

Lab Monitor changes the assessment model from just the final result to the entire development process. We can support and monitor students using AWS AI services.

The system consists of the following parts:

  • A lab monitor agent
  • A lab monitor collector
  • An AR lab assistant

Lab monitor agent

The Lab monitor agent is a Python application that runs on a student’s computer activities. All information is periodically sent to AWS. To identify students and protect the API gateway, each student has a unique API key with a usage limit. The function includes:

  • Capturing all keyboard and pointer events. This can ensure that students are really working on the exercise as it is impossible to complete a coding task without using keyboard and pointer! Also, we encourage students to use shortcuts and we need that information as indicator.
  • Monitoring and controlling PC processes. Teachers can stop students from running programs that are irrelevant to the lab. For computer test, we can kill all browsers and communication software. Processing detailed information is important to decide to upgrade hardware or not!
  • Capturing screens. Amazon Rekognition can detect video or inappropriate content. Extracted text content can trigger an Amazon Sumerian host to talk to a student automatically. It is impossible for a teacher to monitor all student screens! We use a presigned URL with S3 Transfer Acceleration to speed up the image upload.
  • Uploading source code to AWS when students save their code. It is good to know when students complete tasks and to give support to those students who are slower!

Lab monitor collector

The Lab monitor collector is an AWS Serverless Application Model that collects data and provides an API to AR Lab Assistant. Optionally, a teacher can grade students immediately every time they save code by running the unit test inside AWS Lambda. It constantly saves all data into an Amazon S3 data lake and teachers can use Amazon Athena to analyze the data.

To save costs, a scheduled Lambda function checks the teacher’s class calendar every 15 minutes. When there is an upcoming class, it creates a Kinesis stream and Kinesis data analytics application automatically. Teachers can have a nearly real-time view of all student activity.

AR Lab Assistant

The AR lab assistant is a Amazon Sumerian application that reminds students to work on their lab exercise. It sends a camera image to Amazon Rekognition and gets back a student ID.

A Sumerian host, Christine, uses Amazon Polly to speak to students with when something happens:

  • When students pass a unit test, she says congratulations.
  • When students watch movies, she scolds them with the movie actor’s name, such as Tom Cruise.
  • When students watch porn, she scolds them.
  • When students do something wrong, such as forgetting to set up the Python interpreter, she reminds them to set it up.

Students can also ask her questions, for example, checking their overall progress. The host can connect to a Lex chatbot. Student’s conversations are saved in DynamoDB with the sentiment analysis result provided by Amazon Comprehend.

The student screen is like a projector inside the Sumerian application.

Christine: “Stop, watching dirty thing during Lab! Tom Cruise should not be able to help you writing Python code!”

Simplified Architectural Diagrams

Demo video

AR Lab Assistant reaction: https://youtu.be/YZCR2aROBp4

Conclusion

With the combined power of various AWS services, students can now concentrate on only their lab exercise and stop thinking about copying answers from each other! We built the project in about four months and it is still evolving. In a future version, we plan to build a machine learning model to predict the students’ final grade based on their class behavior. They feel that the class is much more fun with Christine.

Lastly, we would like to say thank you to AWS Educate, who provided us with AWS credit, and my AWS Academy student developer team: Mike, Long, Mandy, Tung, Jacqueline, and Hin from IVE Higher Diploma in Cloud and Data Centre Administration. They submitted this application to the AWS Artificial Intelligence (AI) Hackathon and just learned that they received a 3rd place prize!

ProgrammableWebSalesforce Opens Up Lightning Platform to World’s 7 Million+ Javascript Developers

In a move that will open its platform to the more than 7 million worldwide Javascript developers, Salesforce today announced its Lightning Web Components Framework; a technology that makes it possible for developers to use the Javascript programming language to customize browser-based web applications built on top of Salesforce’s core capabilities in the same way they might use Javascript to customize the browser side of any other web app.

ProgrammableWeb: APIsChainFront Cloud Service

The ChainFront Cloud Service is a RESTful API that allows you to create accounts, configure transaction approval methods, create blockchain addresses, and more. This ChainFront API supports JSON format and enables developers with the ability to integrate solutions with blockchain networks like security and user adoption. ChainFront helps developers create useable, scalable, and secure blockchain applications.
Date Updated: 2018-12-13
Tags: Blockchain, Enterprise, , Security, , Transactions

ProgrammableWeb: APIsBandwidth Voice Events

The Bandwidth Voice Events API provides callbacks for Bandwidth Voice Events. Bandwidth sends a message to the application when the call is answered, sends this message to the application when audio file playback is started or done playing etc. This API provides callbacks for answer event, audio file payback, conference events and more. Bandwidth is a communication service that allows users to add voice and SMS services to software and applications that may include; access to local numbers, and scalable pricing, information about an account, a list of transactions made on an account, calls for phone numbers, messages and more.
Date Updated: 2018-12-13
Tags: Telephony, Events

ProgrammableWeb: APIsBandwidth Phone Number

The Bandwidth Phone Number API allows you to search and order phone numbers on demand, searching for phone numbers, porting phone numbers, managing line options, administrative functions and more. The Bandwidth platform provides customers with a simple way of interfacing with the number intelligence platform following standard industry practices with API’s that support REST web service technologies. Bandwidth is a communication service that allows users to add voice and SMS services to software and applications that may include; access to local numbers, and scalable pricing, information about an account, a list of transactions made on an account, calls for phone numbers, messages and more.
Date Updated: 2018-12-13
Tags: Telephony, Search

ProgrammableWeb: APIsClearIP

Build location-aware website and mobile apps with Clear IP API. Locate your users, enrich their experiences, prevent fraud, ensure compliance, and so much more. Clear IP provides IP Address location data such as city, country, geocoordinates, flag, languages, borders and more. Clear IP is provided by PushBots inc.
Date Updated: 2018-12-13
Tags: Location, Localization

ProgrammableWeb: APIsPayCore Public

The PayCore Public API is a payment infrastructure-as-a-service that follows the JSON:API specification. Developers can access payment providers, payment methods, currencies, customer information, rate schemes, payment schemes, and more. The API integrates payment providers from around the world into one hub in order to offer different payment options such as credit cards, e-wallets, and bank transfers. The Public API requires API Keys for authentication and requires the use of the JSON:API media type (application/vnd.api+json) for exchanging data.
Date Updated: 2018-12-13
Tags: Financial, Infrastructure-as-a-Service, , Monetization, , Payments, , Transactions

ProgrammableWeb: APIsAxesso Amazon Product Lookup

The Axesso Amazon Product Lookup API returns detailed Amazon product information including product title, manufacturer, reviews, size selection, and retail price. Additionally, the API supports product recommendations as provided by Amazon. This service is comparable to the Amazon Product Advertising API, but presented in a more condensed manner. The API is free to use and requires Tokens in production mode.
Date Updated: 2018-12-13
Tags: eCommerce, Data-as-a-Service

ProgrammableWeb: APIsNotificare Live

The Notificare Live API utilizes webhooks to integrate real-time functionalities such as Geo-triggers and notifications with third party applications. The API supports public events, product purchases, user profile changes, device property changes, and import job changes. JSON is the preferred response format.
Date Updated: 2018-12-13
Tags: Notifications, Marketing

ProgrammableWebGoogle+ Going Away Sooner than Expected

The consumer version of Google+ is going away sooner than originally planned. In October of this year, Google announced its plans to shut down the consumer version after low use and API vulnerabilities. Google intended to give users 10 months to download data and migrate away from the service. Now, another data leak was discovered, and Google will end access in April 2019.

ProgrammableWebHow Non-Coders Can Get Started with API Trading

Automation is becoming prevalent in the trading community due to the benefits it provides traders, among them the ability to reduce human error while trading as well as the ability to customize trading strategies beyond previous standards. Automated trading involves creating a system to parse incoming market data and apply a set of user-defined rules to make trading decisions based on the incoming data.

Simon Willison (Django)nip.io

nip.io

"NIP.IO maps <anything>.<IP Address>.nip.io to the corresponding <IP Address>, even 127.0.0.1.nip.io maps to 127.0.0.1" - looks useful. xip.io is a different service that does the same thing. Being able to put anything at the start looks handy for testing systems that handle different subdomains.

Simon Willison (Django)The _repr_html_ method in Jupyter notebooks

The _repr_html_ method in Jupyter notebooks

Today I learned that if you add a _repr_html_ method returning a string of HTML to any Python class Jupyter notebooks will render that HTML inline to represent that object.

Via A Gist by Sam Liu

Amazon Web ServicesNow Open – AWS Europe (Stockholm) Region

The AWS Region in Sweden that I promised you last year is now open and you can start using it today! The official name is Europe (Stockholm) and the API name is eu-north-1. This is our fifth region in Europe, joining the existing regions in Europe (Ireland), Europe (London), Europe (Frankfurt), and Europe (Paris). Together, these regions provide you with a total of 15 Availability Zones and allow you to architect applications that are resilient and fault tolerant. You now have yet another option to help you to serve your customers in the Nordics while keeping their data close to home.

Instances and Services
Applications running in this 3-AZ region can use C5, C5d, D2, I3, M5, M5d, R5, R5d, and T3 instances, and can use of a long list of AWS services including Amazon API Gateway, Application Auto Scaling, AWS Artifact, AWS Certificate Manager (ACM), Amazon CloudFront, AWS CloudFormation, AWS CloudTrail, Amazon CloudWatch, CloudWatch Events, Amazon CloudWatch Logs, AWS CodeDeploy, AWS Config, AWS Config Rules, AWS Database Migration Service, AWS Direct Connect, Amazon DynamoDB, EC2 Auto Scaling, EC2 Dedicated Hosts, Amazon Elastic Container Service for Kubernetes, AWS Elastic Beanstalk, Amazon Elastic Block Store (EBS), Amazon Elastic Compute Cloud (EC2), Elastic Container Registry, Amazon ECS, Application Load Balancers (Classic, Network, and Application), Amazon EMR, Amazon ElastiCache, Amazon Elasticsearch Service, Amazon Glacier, AWS Identity and Access Management (IAM), Amazon Kinesis Data Streams, AWS Key Management Service (KMS), AWS Lambda, AWS Marketplace, AWS Organizations, AWS Personal Health Dashboard, AWS Resource Groups, Amazon Relational Database Service (RDS), Amazon Aurora, Amazon Route 53 (including Private DNS for VPCs), AWS Server Migration Service, AWS Shield Standard, Amazon Simple Notification Service (SNS), Amazon Simple Queue Service (SQS), Amazon Simple Storage Service (S3), Amazon Simple Workflow Service (SWF), AWS Step Functions, AWS Storage Gateway, AWS Support API, Amazon EC2 Systems Manager (SSM), AWS Trusted Advisor, Amazon Virtual Private Cloud, VM Import, and AWS X-Ray.

Edge Locations and Latency
CloudFront edge locations are already operational in four cities adjacent to the new region:

  • Stockholm, Sweden (3 locations)
  • Copenhagen, Denmark
  • Helsinki, Finland
  • Oslo, Norway

AWS Direct Connect is also available in all of these locations.

The region also offers low-latency connections to other cities and AWS regions in area. Here are the latest numbers:

AWS Customers in the Nordics
Tens of thousands of our customers in Denmark, Finland, Iceland, Norway, and Sweden already use AWS! Here’s a sampling:

Volvo Connected Solutions Group – AWS is their preferred cloud solution provider; allowing them to connect over 800,000 Volvo trucks, buses, construction equipment, and Penta engines. They make heavy use of microservices and will use the new region to deliver services with lower latency than ever before.

Fortum – Their one-megawatt Virtual Battery runs on top of AWS. The battery aggregates and controls usage of energy assets and allows Fortum to better balance energy usage across their grid. This results in lower energy costs and power bills, along with a reduced environmental impact.

Den Norske Bank – This financial services customer is using AWS to provide a modern banking experience for their customers. They can innovate and scale more rapidly, and have devoted an entire floor of their headquarters to AWS projects.

Finnish Rail – They are moving their website and travel applications to AWS in order to allow their developers to quickly experiment, build, test, and deliver personalized services for each of their customers.

And That Makes 20
With today’s launch, the AWS Cloud spans 60 Availability Zones within 20 geographic regions around the world. We are currently working on 12 more Availability Zones and four more AWS Regions in Bahrain, Cape Town, Hong Kong SAR, and Milan.

AWS services are GDPR ready and also include capabilities that are designed to support your own GDPR readiness efforts. To learn more, read the AWS Service Capabilities for GDPR and check out the AWS General Data Protection Regulation (GDPR) Center.

The Europe (Stockholm) Region is now open and you can start creating your AWS resources in it today!

Jeff;

ProgrammableWebThe VA Introduces its First Health API

The US Department of Veterans Affairs (VA) recently announced its first Health API. The goal of the API is to empower the next generation of web and mobile apps that allow Veterans to interact with their personal health data.

Simon Willison (Django)Things About Real-World Data Science Not Discussed In MOOCs and Thought Pieces

Things About Real-World Data Science Not Discussed In MOOCs and Thought Pieces

Really good article, pointing out that carefully optimizing machine learning models is only a small part of the day-to-day work of a data scientist: cleaning up data, building dashboards, shipping models to production, deciding on trade-offs between performance and production and considering the product design and ethical implementations of what you are doing make up a much larger portion of the job.

Via minimaxir

Amazon Web ServicesAnd Now a Word from Our AWS Heroes…

Whew! Now that AWS re:Invent 2018 has wrapped up, the AWS Blog Team is taking some time to relax, recharge, and to prepare for 2019.

In order to wrap up the year in style, we have asked several of the AWS Heroes to write guest blog posts on an AWS-related topic of their choice. You will get to hear from Machine Learning Hero Cyrus Wong (pictured at right), Community Hero Markus Ostertag, Container Hero Philipp Garbe, and several others.

Each of these Heroes brings a fresh and unique perspective to the AWS Blog and I know that you will enjoy hearing from them. We’ll have the first post up in a day or two, so stay tuned!

Jeff;

ProgrammableWebGoogle Updates ARCore and Sceneform So You Can Make More Realistic Worlds

Google wants your augmented reality environment to look good. The key to AR's magic, says Google, is the way it merges the real with the unreal. In order to truly transport AR users into a new experience, objects and spaces need to look as realistic as possible. That's why the company has updated ARCore and Sceneform

ProgrammableWebMicrosoft Open Sourcing Edge Development through Chromium Project

Microsoft recently announced it's taking an open source approach to the development of its web browser: Microsoft Edge. Interestingly, it's open source path for the browser is the Chromium open source project (the Google-started project that remains the core codebase for the Google Chrome web browser).

ProgrammableWeb: APIsVeterans Affairs Health

The Veterans Affairs Health API provides access to Health data that includes services for; Allergy Intolerance, Condition, Diagnostic Reports and more. The VA Health Information APIs support industry standards Fast Healthcare Interoperability Resources [FHIR]) and provides access to healthcare data via a EHR-agnostic RESTful web services abstraction layer. The FHIR specification defines a set of "Resources" that represent granular clinical concepts. This service is compliant with the FHIR Argonaut Data Query Implementation Guide.
Date Updated: 2018-12-10
Tags: Health, Compliance, , Data, , Government, , Healthcare, , Medical Records

ProgrammableWeb: APIsNomics Crypto Market Data

The Nomics API is an enterprise-grade developer tool for funds, family offices, enterprises, and fintech developers. Our API enables programmatic access to clean, normalized, and gapless primary source trade data across a number of cryptocurrency exchanges.  Instead of having to integrate with multiple exchange APIs of varying quality, you can get everything through one screaming fast fire hose. If you found that you or your developer have to spend too much time cleaning up and maintaining datasets, instead of identifying opportunities, or if you’re tired of interpolated data and want raw primary source trades delivered simply and consistently with top-notch support and SLAs, then check us out here. One notable difference between Nomics’ API and competition is that the Nomics product has no rate limits.
Date Updated: 2018-12-10
Tags: Cryptocurrency, Financial

ProgrammableWebDaily API RoundUp: Purk, TravelTime, Leapwork, Stellar

Every day, the ProgrammableWeb team is busy, updating its three primary directories for APIs, clients (language-specific libraries or SDKs for consuming or providing APIs), and source code samples.

Daniel Glazman (Disruptive Innovations)Edge and Chromium, a different analysis

I am quite surprised by all the public reactions I read about Microsoft's last browser moving to Chromium. I think most if not all commenters have missed the real point, a real point that seems to me way bigger than Edge. Even Mozilla's CEO Chris Beard has not mentioned it. People at Microsoft must be smiling and letting go loud french « Ahlala...». Let me remind everyone that a browser is, from a corporate point of view,  a center of cost and not a center of revenue; if you're really nitpicking, you can call it a center of indirect revenue. So let's review and analyse the facts:

  • I am surprised by the codename supposedly attached to that future version of Microsoft's browser, Anaheim. That codename is not confirmed by Microsoft but I find it quite surprising for a web browser... First, Anaheim is in California and not in Washington State where most of the browser stuff is supposed to happen; yes, it's a detail but still, it's a surprising one. Secondly, Anaheim is really a weird codename in the history of browser codenames at Microsoft. So what happened in Anaheim, CA? A decisive meeting?
  • The blog article about Edge and Chromium was published by a Corporate Vice President of the Windows division. That's absolutely not normal for a browser-only decision.
  • Edge's and IE's market share are, sorry to my dear Microsoft friends, not enough to care that much about such a change. Yes, the browser ecosystem is like a real ecosystem and the lack of genetic diversity that implies EdgeHTML's retirement (see also immediately below) is a global concern. But from a business point of view, nothing to see here, sorry.
  • The blog article and the Github readme page (most people have not seen that one...) say Edge will switch to Chromium. They don't say that EdgeHTML will die. As a matter of fact, EdgeHTML itself is mentioned in the blog article's title and only there, and not at all in the GH page.
  • Microsoft's CEO is currently impulsing a change that sounds to me like a new Samsung's « Change everything but your wife and children ». The tech debt at Microsoft is immense and Nadella rang the rush bell.

So I think the whole thing is not about Edge. The microcosm reacted, and reacted precisely as expected (again, probable laughters in Redmond), but this is really about Windows and the core of activity of Microsoft. Impulsing a change like a move to Chromium and using it as a public announcement by a Windows CVP, is, beyond technical and business choices, a political signal. It says « expect the unexpected ».

I think Microsoft Windows as we know it is about to change and change drastically. Windows as we know it could even die and Microsoft move to another new, different operating system, Edge+Chromium's announcement being only the top of the iceberg. And it's well known that 9/10th of an iceberg remain below water surface.

The gravity center of the company is then about to change too; Nadella probably knows too well the impact of the Windows division on the rest of the company during the Vista years and he certainly knows too well the inter-division wars at Microsoft. It could be highly time to shake the whole thing. As I told Dean Hachamovitch long ago, « you need a commando and what you have now is a mexican army with a lot of generals and not enough soldiers ». Still valid?

Of course, I could be partially or even totally wrong. But I don't think so. This announcement is weird on too many counts, and it's most certainly on purpose. It seems to be telling us « guys, read between the lines, the big message is right there ».

ProgrammableWebPicasa Web Albums Data API to be Deprecated

In January 2019, Google is deprecating its Picasa Web Albums Data API and disabling all associated OAuth scopes. Any existing applications using the API will continue to work until that time. In March 2019, the Picasa Web Albums API will be turned off at which point all calls will fail.

Amazon Web ServicesLearn about New AWS re:Invent Launches – December AWS Online Tech Talks

AWS Tech Talks

Join us in the next couple weeks to learn about some of the new service and feature launches from re:Invent 2018. Learn about features and benefits, watch live demos and ask questions! We’ll have AWS experts online to answer any questions you may have. Register today!

Note – All sessions are free and in Pacific Time.

Tech talks this month:

Compute

December 19, 2018 | 01:00 PM – 02:00 PM PTDeveloping Deep Learning Models for Computer Vision with Amazon EC2 P3 Instances – Learn about the different steps required to build, train, and deploy a machine learning model for computer vision.

Containers

December 11, 2018 | 01:00 PM – 02:00 PM PTIntroduction to AWS App Mesh – Learn about using AWS App Mesh to monitor and control microservices on AWS.

Data Lakes & Analytics

December 10, 2018 | 11:00 AM – 12:00 PM PTIntroduction to AWS Lake Formation – Build a Secure Data Lake in Days – AWS Lake Formation (coming soon) will make it easy to set up a secure data lake in days. With AWS Lake Formation, you will be able to ingest, catalog, clean, transform, and secure your data, and make it available for analysis and machine learning.

December 12, 2018 | 11:00 AM – 12:00 PM PTIntroduction to Amazon Managed Streaming for Kafka (MSK) – Learn about features and benefits, use cases and how to get started with Amazon MSK.

Databases

December 10, 2018 | 01:00 PM – 02:00 PM PTIntroduction to Amazon RDS on VMware – Learn how Amazon RDS on VMware can be used to automate on-premises database administration, enable hybrid cloud backups and read scaling for on-premises databases, and simplify database migration to AWS.

December 13, 2018 | 09:00 AM – 10:00 AM PTServerless Databases with Amazon Aurora and Amazon DynamoDB – Learn about the new serverless features and benefits in Amazon Aurora and DynamoDB, use cases and how to get started.

Enterprise & Hybrid

December 19, 2018 | 11:00 AM – 12:00 PM PTHow to Use “Minimum Viable Refactoring” to Achieve Post-Migration Operational Excellence – Learn how to improve the security and compliance of your applications in two weeks with “minimum viable refactoring”.

IoT

December 17, 2018 | 11:00 AM – 12:00 PM PTIntroduction to New AWS IoT Services – Dive deep into the AWS IoT service announcements from re:Invent 2018, including AWS IoT Things Graph, AWS IoT Events, and AWS IoT SiteWise.

Machine Learning

December 10, 2018 | 09:00 AM – 10:00 AM PTIntroducing Amazon SageMaker Ground Truth – Learn how to build highly accurate training datasets with machine learning and reduce data labeling costs by up to 70%.

December 11, 2018 | 09:00 AM – 10:00 AM PTIntroduction to AWS DeepRacer – AWS DeepRacer is the fastest way to get rolling with machine learning, literally. Get hands-on with a fully autonomous 1/18th scale race car driven by reinforcement learning, 3D racing simulator, and a global racing league.

December 12, 2018 | 01:00 PM – 02:00 PM PTIntroduction to Amazon Forecast and Amazon Personalize – Learn about Amazon Forecast and Amazon Personalize – what are the key features and benefits of these managed ML services, common use cases and how you can get started.

December 13, 2018 | 01:00 PM – 02:00 PM PTIntroduction to Amazon Textract: Now in Preview – Learn how Amazon Textract, now in preview, enables companies to easily extract text and data from virtually any document.

Networking

December 17, 2018 | 01:00 PM – 02:00 PM PTIntroduction to AWS Transit Gateway – Learn how AWS Transit Gateway significantly simplifies management and reduces operational costs with a hub and spoke architecture.

Robotics

December 18, 2018 | 11:00 AM – 12:00 PM PTIntroduction to AWS RoboMaker, a New Cloud Robotics Service – Learn about AWS RoboMaker, a service that makes it easy to develop, test, and deploy intelligent robotics applications at scale.

Security, Identity & Compliance

December 17, 2018 | 09:00 AM – 10:00 AM PTIntroduction to AWS Security Hub – Learn about AWS Security Hub, and how it gives you a comprehensive view of high-priority security alerts and your compliance status across AWS accounts.

Serverless

December 11, 2018 | 11:00 AM – 12:00 PM PTWhat’s New with Serverless at AWS – In this tech talk, we’ll catch you up on our ever-growing collection of natively supported languages, console updates, and re:Invent launches.

December 13, 2018 | 11:00 AM – 12:00 PM PTBuilding Real Time Applications using WebSocket APIs Supported by Amazon API Gateway – Learn how to build, deploy and manage APIs with API Gateway.

Storage

December 12, 2018 | 09:00 AM – 10:00 AM PTIntroduction to Amazon FSx for Windows File Server – Learn about Amazon FSx for Windows File Server, a new fully managed native Windows file system that makes it easy to move Windows-based applications that require file storage to AWS.

December 14, 2018 | 01:00 PM – 02:00 PM PTWhat’s New with AWS Storage – A Recap of re:Invent 2018 Announcements – Learn about the key AWS storage announcements that occurred prior to and at re:Invent 2018. With 15+ new service, feature, and device launches in object, file, block, and data transfer storage services, you will be able to start designing the foundation of your cloud IT environment for any application and easily migrate data to AWS.

December 18, 2018 | 09:00 AM – 10:00 AM PTIntroduction to Amazon FSx for Lustre – Learn about Amazon FSx for Lustre, a fully managed file system for compute-intensive workloads. Process files from S3 or data stores, with throughput up to hundreds of GBps and sub-millisecond latencies.

December 18, 2018 | 01:00 PM – 02:00 PM PTIntroduction to New AWS Services for Data Transfer – Learn about new AWS data transfer services, and which might best fit your requirements for data migration or ongoing hybrid workloads.

Amazon Web ServicesNew – EC2 P3dn GPU Instances with 100 Gbps Networking & Local NVMe Storage for Faster Machine Learning + P3 Price Reduction

Late last year I told you about Amazon EC2 P3 instances and also spent some time discussing the concept of the Tensor Core, a specialized compute unit that is designed to accelerate machine learning training and inferencing for large, deep neural networks. Our customers love P3 instances and are using them to run a wide variety of machine learning and HPC workloads. For example, fast.ai set a speed record for deep learning, training the ResNet-50 deep learning model on 1 million images for just $40.

Raise the Roof
Today we are expanding the P3 offering at the top end with the addition of p3dn.24xlarge instances, with 2x the GPU memory and 1.5x as many vCPUs as p3.16xlarge instances. The instances feature 100 Gbps network bandwidth (up to 4x the bandwidth of previous P3 instances), local NVMe storage, the latest NVIDIA V100 Tensor Core GPUs with 32 GB of GPU memory, NVIDIA NVLink for faster GPU-to-GPU communication, AWS-custom Intel® Xeon® Scalable (Skylake) processors running at 3.1 GHz sustained all-core Turbo, all built atop the AWS Nitro System. Here are the specs:4

Model NVIDIA V100 Tensor Core GPUs GPU Memory NVIDIA NVLink vCPUs Main Memory Local Storage Network Bandwidth EBS-Optimized Bandwidth
p3dn.24xlarge 8 256 GB 300 GB/s 96 768 GiB 2 x 900 GB NVMe SSD 100 Gbps 14 Gbps

If you are doing large-scale training runs using MXNet, TensorFlow, PyTorch, or Keras, be sure to check out the Horovod distributed training framework that is included in the Amazon Deep Learning AMIs. You should also take a look at the new NVIDIA AI Software containers in the AWS Marketplace; these containers are optimized for use on P3 instances with V100 GPUs.

With a total of 256 GB of GPU memory (twice as much as the largest of the current P3 instances), the p3dn.24xlarge allows you to explore bigger and more complex deep learning algorithms. You can rotate and scale your training images faster than ever before, while also taking advantage of the Intel AVX-512 instructions and other leading-edge Skylake features. Your GPU code can scale out across multiple GPUs and/or instances using NVLink and the NVLink Collective Communications Library (NCCL). Using NCCL will also allow you to fully exploit the 100 Gbps of network bandwidth that is available between instances when used within a Placement Group.

In addition to being a great fit for distributed machine learning training and image classification, these instances provide plenty of power for your HPC jobs. You can render 3D images, transcode video in real time, model financial risks, and much more.

You can use existing AMIs as long as they include the ENA, NVMe, and NVIDIA drivers. You will need to upgrade to the latest ENA driver to get 100 Gbps networking; if you are using the Deep Learning AMIs, be sure to use a recent version that is optimized for AVX-512.

Available Today
The p3dn.24xlarge instances are available now in the US East (N. Virginia) and US West (Oregon) Regions and you can start using them today in On-Demand, Spot, and Reserved Instance form.

Bonus – P3 Price Reduction
As part of today’s launch we are also reducing prices for the existing P3 instances. The following prices went in to effect on December 6, 2018:

  • 20% reduction for all prices (On-Demand and RI) and all instance sizes in the Asia Pacific (Tokyo) Region.
  • 15% reduction for all prices (On-Demand and RI) and all instance sizes in the Asia Pacific (Sydney), Asia Pacific (Singapore), and Asia Pacific (Seoul) Regions.
  • 15% reduction for Standard RIs with a three-year term for all instance sizes in all regions except Asia Pacific (Tokyo), Asia Pacific (Sydney), Asia Pacific (Singapore), and Asia Pacific (Seoul).

The percentages apply to instances running Linux; slightly smaller percentages apply to instances that run Microsoft Windows and other operating systems.

These reductions will help to make your machine learning training and inferencing even more affordable, and are being brought to you as we pursue our goal of putting machine learning in the hands of every developer.

Jeff;

 

 

ProgrammableWebSignEasy API Enables In-App Document Signing

SignEasy, eSignature solution provider, has announced the SignEasy API. Through the API, developers can integrate SignEasy technology into any document workflow that requires customer signatures. Once integrated, any app or website can include document signing as an automated feature.

ProgrammableWebGoogle Provides New Tools for Live Streaming Video

Earlier this week Google announced new tools that aim to make it easier for content providers to share when they are streaming live video.

ProgrammableWeb: APIsFreshProxies

The FreshProxies API provides access to the frequently and thoroughly tested public proxies. Selecting by countries filter. Ordering by uptime, speed, success rate and freshness.
Date Updated: 2018-12-07
Tags: Data Mining, Networking, , Security

ProgrammableWebGoogle Replacing Drive Android API with Drive REST API

The Google Drive API makes it possible for developers to add Google Drive file access to their own apps. Well, Google plans to retire the Google Drive Android API next year and replace it with the Google Drive REST API. Get ready to migrate your apps before December 6, 2019. 

ProgrammableWebMicrosoft Introduces Azure API Management for Serverless Environments

Microsoft's Azure API Management is a service that allows customers to publish, secure, transform, maintain and monitor APIs. Through the service, developers can create programmatic access between two different applications and/or services. API Management handles many common API tasks including mediating calls, authentication, authorization, rate limiting, quota enforcement requests, logging, tracing and version management.

ProgrammableWeb: APIsSignEasy

The SignEasy API is a Rest service that allows you to integrate the eSigning experience into your application or business workflow. This API supports JSON and enables you to retrieve user details, delete a signed file, create a signature request that includes; User, Originals, Signed completed and more. SignEasy provides a way to sign and get documents signed from smartphones, tablets and the web.
Date Updated: 2018-12-06
Tags: Electronic Signature, Documents

ProgrammableWeb: APIsAlpaca Trade Streaming

The Alpaca Trade Streaming API provides your Alpaca trading algorithm, with access to real-time price and fundamentals. It is the ability to place orders and manage your portfolio, with a streaming style that listens to one or more streams, and allows you to get updates on account and trade changes. WebSocket clients can listen to trade streams real-time data for any Alpaca accounts. This API can be authenticated using your Alpaca API key that must be sent with an authentication message over the WebSocket connection as its payload. Alpaca API provides a way to build and trade with real-time market data for free.
Date Updated: 2018-12-06
Tags: Stocks, Data, , Real Time, , Streaming

ProgrammableWeb: APIsClimBiz ADAPT Climate Projection

The ClimBiz ADAPT Climate Project API allows users to access climate projection information for different scenarios, data types (i.e. temperature, precipitation, etc), and geographies such as countries, states/provinces, etc.
Date Updated: 2018-12-06
Tags: Climate, Predictions

ProgrammableWebSecurity Hole Exposes Kubernetes

Kubernetes, one of the most popular cloud container orchestration systems, has been hit by a security flaw. The bug, now fixed, is called VCE-2018-1002105 and was deemed a critical security hole. Although the fix is in, users must upgrade to the latest Kubernetes version, which isn't always easy for administrators. 

Henry Sivonenencoding_rs: a Web-Compatible Character Encoding Library in Rust

encoding_rs is a high-decode-performance, low-legacy-encode-footprint and high-correctness implementation of the WHATWG Encoding Standard written in Rust.

Footnotes

Updated: .  Michael(tm) Smith <mike@w3.org>