Daniel Glazman (Disruptive Innovations)Bulgaria Web Summit

I will be speaking at the Bulgaria Web Summit 2015 in Sofia, Bulgaria, 18th of may.

Norman Walsh (Sun)XInclude 1.1 (Last Call mark II)

<article class="essay" id="content" lang="en">

Implementation experience, there's nothing like it.

Some time back, we got XInclude 1.1 wrapped up. It was all done but for the implementations.

XInclude 1.1, provides a few small extensions:

  • It allows non-XML fragment identifiers so that, for example, fragments of text/plain documents can be included.

  • It adds the ability to copy attributes into the included content; this allows authors to add annotations that allow a post-processing step to identify the XInclude boundaries. Identifying the boundaries allows such a step to perform other processing, such as ID/IDREF fixup.

  • Licenses implementations to provide support for additional fragment identifier schemes.

All well and good. The annotation mechanism was simple: any namespace qualified attributes on the <xi:include> element would be copied to the (top-level) included elements[1].

As ever, the struggle was between simplicity and expressive power. “Copy namespace qualified attributes” is simple. Its consequences: there's no way to annotate with attributes that aren't in a namespace and it copies the xml:* attributes, which has pros and cons. Pro: copying xml:id is definitely a “works for the simplest cases” duplicate ID fixup mechanism. Con: copying the xml:lang attribute is most likely to do the wrong thing.

And then there's xml:base. Implementation experience taught us that copying xml:base naïvely did the wrong thing. And doing the right thing was…complicated.

Live and learn.

So we're going back to Last Call (with a plan to go stright to PR once we have implementations). Here's what we changed.

  • We don't copy the xml:* attributes anymore. We still copy all other namespace qualified attributes.

  • There's a new XInclude attribute, set-xml-id (n.b. not in a namespace). The value specified is used as the value for the xml:id attribute on top-level included elements.

  • Forced to re-open the can of worms, we accepted the request to provide some mechanism to support annotations not in a namespace. Attributes in the “http://www.w3.org/2001/XInclude/local-attributes” namespace will be copied to attributes with no namespace.

    In other words, <xi:include xila:class="highlight" href="…"/> will annotate top-level elements with a class attribute (in no namespace, with the value “highlight”).

Share and enjoy. And implement, please.


[1]This is not a 100% solution, but it's well over 80%.

</article>

Daniel Glazman (Disruptive Innovations)Logo sur MO5/TO7/TO9, une tranche de vie (de geek)

À cause de Samy Rabih sur twitter, on vient de me rappeler un épisode plutôt sympa de ma vie de geek... Quelques jours après le bac, mes profs de physique et maths de terminale, Patrick Foulon (désormais directeur du Centre International de Rencontres Mathématiques à Marseille, cordial salut à lui s'il lit ces lignes) et Francis Rabany (un pote, un type en or avec un coeur grand comme ça, malheureusement décédé en 1997), m'ont contacté à cause de ma très forte tendance à occuper la salle d'informatique de l'école pendant toutes mes heures libres. Enfin presque toutes mes heures puisque je m'occupais aussi du Labo Photo de l'école avec la très charmante et très délicieuse Catherine Leclere-Bessonnet. Réunion avec eux dans des locaux derrière le périph Porte d'Orléans. En gros, ils montaient une boîte de logiciels éducatifs avec Belin et me proposaient d'écrire from scratch, pour le plan Thomson MO5/TO7/TO9 de l'Éducation Nationale, un langage LOGO complet...

J'ai commencé à bosser dessus immédiatement, avec des grands plongeons dans la ROM et le processeur 6809 pour l'accès au crayon optique. J'ai pour la première fois de ma vie passé mes vacances d'été dans les Landes avec une bécane complète (et c'était pas un laptop et y'avait pas de connectivité hein). Rabany et moi passions des heures au téléphone chaque jour pour la définition des fonctionnalités, c'était un vrai bonheur. Le plus pénible était la sauvegarde sur cassette, qui pour éviter le piratage (tu parles!!!) utilisait un décalage en fréquence rendant la recopie de cassette sur cassette impossible sans matériel trafiqué. Enfin, ça ne m'a pas arrêté très longtemps hein :-)

Fin août, l'interpreteur était fini, avec plein de basic et d'assembleur dedans après donc moins de deux mois de développement. Il était très simple, super-lightweight, avec de la vraie récursivité et tout. Fin septembre, il était sous jaquette et vendu à l'EN. Je dois encore avoir une copie d'origine chez mon père...

Si vous avez fait du LOGO à l'école sur un ordinateur Thomson dans les années 80, c'était donc à cause de moi. Désolé :-)

ProgrammableWebAmazon Updates Maps API, Adds 3D Vectoring and Fluid Movement

Amazon recently released Amazon Maps API version 2.0. The mapping API has long enabled developers to add maps to their apps for Amazon’s Fire Tablets and the Fire Phone. The revised API takes Amazon Maps to the next level, thanks to support for 3D vectoring and fluid movement through map environments. 

Amazon Web ServicesResources Groups and Tagging for AWS

For many years, AWS customers have used tags to organize their EC2 resources (instances, images, load balancers, security groups, and so forth), RDS resources (DB instances, option groups, and more), VPC resources (gateways, option sets, network ACLS, subnets, and the like) Route 53 health checks, and S3 buckets. Tags are used to label, collect, and organize resources and become increasingly important as you use AWS in larger and more sophisticated ways. For example, you can tag relevant resources and then take advantage AWS Cost Allocation for Customer Bills.

Today we are making tags even more useful with the introduction of a pair of new features: Resource Groups and a Tag Editor. Resource Groups allow you to easily create, maintain, and view a collection of resources that share common tags. The new Tag Editor allows you to easily manage tags across services and Regions. You can search globally and edit tags in bulk, all with a couple of clicks.

Let's take a closer look at both of these cool new features! Both of them can be accessed from the new AWS menu:

Tag Editor
Until today, when you decided to start making use of tags, you were faced with the task of stepping through your AWS resources on a service-by-service, region-by-region basis and applying tags as needed. The new Tag Editor centralizes and streamlines this process.

Let's say I want to find and then tag all of my EC2 resources. The first step is to open up the Tag Editor and search for them:

The Tag Editor searches my account for the desired resource types across all of the selected Regions and then displays all of the matches:

I can then select all or some of the resources for editing. When I click on the Edit tags for selected button, I can see and edit existing tags and add new ones. I can also see existing System tags:

I can see which values are in use for a particular tag by simply hovering over the Multiple values indicator:

I can change multiple tags simultaneously (changes take effect when I click on Apply changes):

Resource Groups
A Resource Group is a collection of resources that shares one or more tags. It can span Regions and services and can be used to create what is, in effect, a custom console that organizes and consolidates the information you need on a per-project basis.

You can create a new Resource Group with a couple of clicks. I tagged a bunch of my AWS resources with Service and then added the EC2 instances, DB instances, and S3 buckets to a new Resource Group:

My Resource Groups are available from within the AWS menu:

Selecting a group displays information about the resources in the group, including any alarm conditions (as appropriate):

This information can be further expanded:

Each identity within an AWS account can have its own set of Resource Groups. They can be shared between identities by clicking on the Share icon:

Down the Road
We are, as usual, very interested in your feedback on this feature and would love to hear from you! To get in touch, simply open up the Resource Groups Console and click on the Feedback button.

Available Now
Resource Groups and the Tag Editor are available now and you can start using them today!

-- Jeff;

ProgrammableWebGoogle Closes Freebase, Launches Knowlege Graph API

Google is shuttering Freebase, a community-curated database of well-known people, places, and things, and providing a tool that will enable its data to be imported into the Wikidata project. Originally started by a company called Metaweb, Google became Freebase’s owner when the search giant acquired Metaweb in 2010.

In a Google+ post yesterday, the Freebase team explained its decision to hand Freebase data over to Wikidata:

Norman Walsh (Sun)The short-form week of 8–14 Dec 2014

<article class="essay" id="content" lang="en">

The week in review, 140 characters at a time. This week, 3 messages in 7 conversations. (With 3 favorites.)

This document was created automatically from my archive of my Twitter stream. Due to limitations in the Twitter API and occasional glitches in my archiving system, it may not be 100% complete.

Monday at 07:16am

XML Stars, the journal is out! http://t.co/zAYsPMgQ86 Stories via @ndw @james_clark —@dominixml

Tuesday at 09:25pm

FAV
"Our social media campaign went viral!" Congratulations, now take your place alongside influenza, hepatitis, smallpox, herpes and polio.—@tommorris

Wednesday at 12:54pm

FAV
The holocaust was legal, slavery was legal, segregation was legal. If you use the state as a metric for ethics you'll end up disappointed.—@lex_looper

Wednesday at 08:42pm

Thursday at 07:47am

Booked through BA.com, World Traveler Plus: $95. Booked through AA: $784.80. Gee, nice to be platinum on AA again. *grump*—@ndw

Sunday at 05:00pm

Decorating a Christmas tree with all the doors and windows open is ... odd. Nice weather though.—@ndw

Monday at 10:39pm

RT @rands: Hex color of current time: http://t.co/zpDXnbGI5v —@ndw
</article>

Norman Walsh (Sun)The short-form week of 1–7 Dec 2014

<article class="essay" id="content" lang="en">

The week in review, 140 characters at a time. This week, 5 messages in 4 conversations.

This document was created automatically from my archive of my Twitter stream. Due to limitations in the Twitter API and occasional glitches in my archiving system, it may not be 100% complete.

Tuesday at 09:40am

Dear @SignalsCatalog, I'm a former customer. After about four unsuccessful attempts to unsubscribe from your incessant email, I'm done. FYI.—@ndw

In a conversation that started on Sunday at 03:39am

Hotel shower dial turns through 270°. Useful temp between 19.3 and 19.34°. Bonus: occasionally scalding bursts.—@ndw
@ndw are you in the UK by any chance?—@al3xbrown
@al3xbrown I am indeed. Early Christmas with mom. Home on Tuesday. P.S. the mixer tap was invented in the previous millennium.—@ndw

Sunday at 05:16pm

RT @janjarfalk: Inspired by @standupmaths's @numberphile on fractions I decided to render every single fraction known to man. http://t.co/e… —@ndw

Monday at 10:16pm

RT @_TheGeoff: Stretch your arms out. Tip of left hand is Big Bang, tip of right is now. One brush of a nail file would destroy all human h…—@ndw
</article>

Norman Walsh (Sun)The short-form week of 24–30 Nov 2014

<article class="essay" id="content" lang="en">

The week in review, 140 characters at a time. This week, 3 messages in 4 conversations. (With 2 favorites.)

This document was created automatically from my archive of my Twitter stream. Due to limitations in the Twitter API and occasional glitches in my archiving system, it may not be 100% complete.

In a conversation that started on Wednesday at 08:04pm

Achievements unlocked: gingerbread, pumpkin pie, treacle taffy. Achievements on deck: turkey, brussel sprouts, mashed taters, etc.—@ndw
@ndw Sounds good. BTW, next time you see me at something like TPAC, say hi. :)—@bsletten
@ndw I didn’t realize it was you until I was already on the elevator!—@bsletten
@bsletten Aww, drat. I must not have realized. Was distracted and at another hotel. :-/—@ndw

Thursday at 02:20am

Thursday at 08:33am

RT @WyoWeeds: @ramez Is truncating the y-axis dishonest? http://t.co/9tsuVYLxM5 —@ndw

Friday at 04:58pm

FAV
Spectacular night sky seen from Mars. http://t.co/igRbznkyr3 —@isa_MYB
</article>

Norman Walsh (Sun)The short-form week of 17–23 Nov 2014

<article class="essay" id="content" lang="en">

The week in review, 140 characters at a time. This week, 4 messages in 3 conversations.

This document was created automatically from my archive of my Twitter stream. Due to limitations in the Twitter API and occasional glitches in my archiving system, it may not be 100% complete.

In a conversation that started on Sunday at 11:37pm

@JohnTheBastard @gruber @hopsinglaundry Also a shame they no longer sell dry white. Only "extra dry" now in the US. #sadpanda —@ndw
@ndw @JohnTheBastard @hopsinglaundry But that said, it’s not available in Pennsylvania and I’ve never seen it in New Jersey, either.—@gruber
@gruber @ndw I thought both the French and American variety were supposed to be available now, but I haven't specifically looked for them.—@JohnTheBastard
@JohnTheBastard @gruber I haven't seen dry recently. I console myself with the fact that I'm getting 2 bottles of Ambré in December.—@ndw
@ndw @gruber @hopsinglaundry I always preferred the extra dry in martinis (still have half a case of pre-2009 bottles), and Dolin otherwise.—@JohnTheBastard
@ndw @JohnTheBastard @hopsinglaundry Noilly “Extra Dry” is supposedly the same stuff they used to call “Dry French”: http://t.co/vDzjiHDFOq —@gruber
@gruber @JohnTheBastard @hopsinglaundry Perhaps. I still prefer the stuff they used to sell as simply "Dry".—@ndw

In a conversation that started on Tuesday at 01:47am

Enquiring minds want to know: what is the state of the art wrt monads in XQuery? @jpcs @ndw @xquery #fp —@mdubinko
@mdubinko @ndw @xquery Using monads is a style of programming, like OO. You can write in monadic style in XQuery. But there is no IO monad.—@jpcs
@mdubinko @jpcs @ndw 10 yrs ago I read this http://t.co/W9v8XF9Ol6 and this http://t.co/ryHv1JjX8F illustrated monadic io (I get 404 now)—@xquery
@mdubinko @ndw @xquery More importantly, IMO better ways exist to do procedural in functional than confusing monads.—@jpcs
@jpcs @ndw @xquery over in Scala land, I am finding flatMap-able composition a really useful tool. It's monads minus confusion.—@mdubinko
@mdubinko @jpcs @ndw btw we will all be around in Jan ... will give you a shout be great to catch up—@xquery

In a conversation that started on Thursday at 09:38am

@ndw Can you give us a rough estimate when Saxon 9.6 will be supported in @xmlcalabash? We’d like to use the EXPath binary module. Thanks!—@gimsieke
@gimsieke @xmlcalabash Not really. There's a bug in tree construction and it's been a struggle to make a useful test case. #asap —@ndw
@ndw @xmlcalabash Ok, we’ll write a Java extension function for the time being. An add’l advantage to that is that it works with Saxon HE.—@gimsieke
</article>

Amazon Web ServicesEC2 Container Service In Action

We announced the Amazon EC2 Container Service at AWS re:Invent and invited you to join the preview. Since that time, we've seen a lot of interest and a correspondingly high signup rate for the preview. With the year winding down, I thought it would be fun to spend a morning putting the service through its paces. We have already approved all existing requests to join the preview; new requests are currently being approved within 24 hours.

As I noted in my earlier post, this new service will help you to build, run, and scale Docker-based applications. You'll benefit from easy cluster management, high performance, flexible scheduling, extensibility, portability, and AWS integration while running in an AWS-powered environment that is secure and efficient.

Quick Container Review
Before I dive in, let's take a minute to review some of the terminology and core concepts implemented by the Container Service.

  • Cluster - A logical grouping of Container Instances that is used to run Tasks.
  • Container Instance - An EC2 instance that runs the ECS Container Agent and that has been registered into a Cluster. The set of instances running within a Cluster create a pool of resources that can be used to run Tasks.
  • Task Definition - A description of a set of Containers. The information contained in a Task Description defines one or more Containers. All of the Containers defined in a particular Task Definition are run on the same Container Instance.
  • Task - An instantiation of a Task Definition.
  • Container - A Docker container that was created as part of a Task.

The ECS Container Agent runs on Container Instances. It is responsible for starting Containers on behalf of ECS. The agent itself runs within a Docker container (available on Docker Hub) and communicates with the Docker daemon running on the Instance.

When talking about a cluster or container service, "scheduling" refers to the process of assigning tasks to instances. The Container Service provides you with three scheduling options:

  1. Automated - The RunTask function will start a Task (as specified by a Task Definition) on a Cluster using random placement.
  2. Manual - The StartTaskfunction will start a Task (again, as specified by a Task Definition) on a specified Container Instance (or Instances).
  3. Custom - You can use the ListContainerInstances and DescribeContainerInstances functions to gather information about available resources within a Cluster, implement the "brain" of the schedule (in other words, use the available information to choose a suitable Container Instance), and then call StartTask to start a task on the Instance. When you do this you are, in effect, creating your own implementation of RunTask.

EC2 Container Service in Action
In order to gain some first-hand experience with ECS, I registered for the preview and then downloaded, installed, and configured a preview version of the AWS CLI. Then I created an IAM Role and a VPC and set about to create my cluster (ECS is currently available in US East (Northern Virginia) with support for other Regions expected in time). I ran the following command:

$ aws ecs create-cluster --cluster-name MyCluster --profile jbarr-cli

The command returned information about my new cluster as a block of JSON:

{
    "cluster": {
        "clusterName": "MyCluster", 
        "status": "ACTIVE", 
        "clusterArn": "arn:aws:ecs:us-east-1:348414629041:cluster/MyCluster"
    }
}

Then I launched a couple of EC2 instances into my VPC using an ECS-enabled AMI that had been shared with me as part of the preview process (this is a very lightweight version of the Amazon Linux AMI, optimized and tuned for ECS). I chose my new IAM Role (ecs) as part of the launch process:

I also edited the instance's User Data to make the instance launch in to my cluster:

After the instances launched I was able to see that they were part of my cluster:

$ aws ecs list-container-instances --cluster MyCluster --profile jbarr-cli
{
    "containerInstanceArns": [
        "arn:aws:ecs:us-east-1:348414629041:container-instance/4cf62484-da62-49a5-ad32-2015286a6d39", 
        "arn:aws:ecs:us-east-1:348414629041:container-instance/be672053-0ff8-4478-b136-7fae9225e493"
    ]
}

I can choose an instance and query it to find out more about the registered and available CPU and memory resources:

$ aws ecs describe-container-instances --cluster MyCluster \
  --container-instances arn:aws:ecs:us-east-1:348414629041:container-instance/4cf62484-da62-49a5-ad32-2015286a6d39 \
  --profile jbarr-cli

Here's an excerpt from the returned data:

{
            "registeredResources": [
                {
                    "integerValue": 1024, 
                    "longValue": 0, 
                    "type": "INTEGER", 
                    "name": "CPU", 
                    "doubleValue": 0.0
                }, 
                {
                    "integerValue": 3768, 
                    "longValue": 0, 
                    "type": "INTEGER", 
                    "name": "MEMORY", 
                    "doubleValue": 0.0
                }
            ]
}

Following the directions in the Container Service Developer Guide, I created a simple task definition and registered it:

$ aws ecs register-task-definition --family sleep360 \
  --container-definitions file://$HOME/tmp/task.json \
  --profile jbarr-cli

Then I ran 10 copies of the task:

aws ecs run-task --cluster MyCluster --task-definition sleep360:1 --count 10 --profile jbarr-cli

And I listed the running tasks:

$ aws ecs list-tasks --cluster MyCluster --profile jbarr-cli

This is what I saw:

{
    "taskArns": [
        "arn:aws:ecs:us-east-1:348414629041:task/0c949733-862c-4979-b5bd-d4f8b474c58e", 
        "arn:aws:ecs:us-east-1:348414629041:task/3ababde9-08dc-4fc9-b005-be5723d1d495", 
        "arn:aws:ecs:us-east-1:348414629041:task/602e13d2-681e-4c87-a1d9-74c139f7335e", 
        "arn:aws:ecs:us-east-1:348414629041:task/6d072f42-75da-4a84-8b68-4841fdfe600d", 
        "arn:aws:ecs:us-east-1:348414629041:task/6da6c947-8071-4111-9d31-b87b8b93cc53", 
        "arn:aws:ecs:us-east-1:348414629041:task/6ec9828a-cbfb-4a39-b491-7b7705113ad2", 
        "arn:aws:ecs:us-east-1:348414629041:task/87e29ab2-34be-4495-988b-c93ac1f8b77c", 
        "arn:aws:ecs:us-east-1:348414629041:task/ad4fc3cc-7e80-4681-b858-68ff46716fe5", 
        "arn:aws:ecs:us-east-1:348414629041:task/cdd221ea-837c-4108-9577-2e4f53376c12", 
        "arn:aws:ecs:us-east-1:348414629041:task/eab79263-087f-43d3-ae4c-1a89678c7101"
    ]
}

I spent some time describing the tasks and wrapped up by shutting down the instances. After going through all of this (and making a mistake or two along the way due to being so eager to get a cluster up and running), I'll leave you with three simple reminders:

  1. Make sure that your VPC has external connectivity enabled.
  2. Make sure to use the proper, ECS-enabled AMI.
  3. Make sure to launch the AMI with the requisite IAM Role.

ECS Quickstart Template
We have created an ECS Quickstart Template for CloudFormation to help you to get up and running even more quickly. The template creates an IAM Role and an Instance Profile for the Role. The Role supplies the permission that allows the ECS Agent to communicate with ECS. The template launches an instance using the Role and returns an SSH command that can be used to access the instance. You can launch the instance in to an existing cluster, or you can use the name "default" to create (if necessary) a default cluster. The instance is always launched within your Default VPC.

Contain Yourself
If you would like to get started with ECS, just register now and we'll get you up and running as soon as possible.

To learn more about ECS, spend 30 minutes watching this session from re:Invent (one caveat: the video is already a bit dated; for example, Task Definitions are no longer versioned):

You can also register for our upcoming (January 14th, 2015) webinar, Amazon EC2 Container Service Deep Dive. In this webinar, my colleague Deepak Singh will will talk about why we built EC2 Container Service, explain some of the core concepts, and show you how to use the service for your applications.

CoreOS is a new Linux distribution designed to support the needs of modern infrastructure stacks. The CoreOS AMI now supports ECS; you can read the Amazon ECS on CoreOS documentation to learn more.

As always, we are interested in your feedback. With ECS still in preview mode, now is the perfect time for you to let us know more about your needs. You can post your feedback to the ECS Forum. you can also create AWS Support cases if you are in need of assistance.

-- Jeff;

ProgrammableWebMigme&#039;s Developer Program Monetizes Emerging Markets

Migme, a social entertainment platform focused on emerging markets, has launched a developer program that gives developers access to the platform's functionality, as well as its growing user base. The APIs give full access, opening up the opportunities to innovate further and to tap into new markets to grow revenue.
 

ProgrammableWebLibscore Aims to Measure Popularity of JavaScript Libraries

Libscore, a brand new tool that explores the popularity of third-party JavaScript libraries, was just announced by the project's sponsors Stripe and DigitalOcean.

ProgrammableWebSinch&#039;s New SMS API Reaches Over 100 Countries

Sinch, a cloud-based communications platform, has launched a new SMS API that enables apps to send SMS messages to users in over 100 countries. The SMS API adds to Sinch’s existing communications API portfolio that included IM and voice functionality.

Matt Webb (Schulze & Webb)Filtered for nematodes and Uniqlo

1.

The nematode worm Caenorhabditis elegans (C. elegans) is tiny and only has 302 neurons. These have been completely mapped and the OpenWorm project is working to build a complete simulation of the worm in software. A neuron map is called a connectome.

(In May, OpenWorm achieved a successful Kickstarter to run the worm's brain in a web browser.)

One of the OpenWorm founders has hooked up the software connectome to sensors and wheels in a Lego robot body: A Worm's Mind In A Lego Body.

It's a funny threshold to cross without much fanfare, the first brain upload.

Is this Artificial Intelligence? What's A. about this A.I.?

Related: The four-colour theorem which was the first to rely on Proof By Computer. Instead of being solved mathematically, every single of the vast number of cases was checked by a computer program. Does this count as a proof? Controversial at the time, more common now.

Related: Slime mold robot. Related: Cyborg cockroaches.

Caenorhabditis elegans. C. elegans.

Is the uploaded nematode a new species? If so, what do we call it?

2.

Books are back in the UK. E-book sales have peaked at 30%; Waterstones (major but recently troubled chain bookshop) is beginning to open new branches.

Old books undergo acid hydrolosis -- lignin, which binds the fibres, oxidises into acids which break down the cellulose. The organic compounds released smell of vanilla and almonds.

3.

I mentioned the Cereal Killer cafe the other day... here's another perspective:

Cereal Killer Cafe, the London Review of Breakfasts.

A wave of nausea suddenly hits me. I'm staring at my notes and the room feels like it's breathing. Then the rest just pours out. 'Is your cafe ironic? Do you really like ADHD kids food? Or just jokingly like it? Is there really anything to celebrate here beyond a profound efficiency in the delivery of deadly consumption habit forming food to minors? Or is that the point?'

Must read.

4.

I get a few specific items from particular brands. For the basics, I love Uniqlo. Great clothes.

...maybe a little bit because I'm in love with their slogans, which every employee must memorise, as related in this GQ article on Uniqlo.

Uniqlo is clothes that suit your values.

Uniqlo is how the future dresses.

CHANGE OR DIE

Fast Company on Uniqlo: We are not a fashion company, ... We are a technology company.

At the factory, a technician hands me a packet of small white pellets that look like albino peppercorns. These are the seeds of HeatTech.

Uniqlo is beauty in hyperpracticality.

Uniqlo is clothing in the absolute.

In Snacks for a Fat Planet (New Yorker, 2011), it turns out that PepsiCo have invented a new kind of salt. So we wondered, was there a different kind of salt crystal that would produce the same taste curve but with less salt?

Yes: 'We don’t know the molecular structure of the salt receptors, and we don't really understand the mechanism by which salt works,' Khan went on. Nevertheless, collaborating with crystal technologists in Munich, PepsiCo was able to develop '15 micron salt,' a new kind of salt that produces the same taste curve as the salt the company has been using - a pyramid-shaped crystal known as Alberger salt - but contains twenty-five to forty per cent less sodium. PepsiCo first used the new salt on its Walker brand of chips, which it sells in the U.K. By the end of 2012, 15 micron salt will be flavoring many of the Lay's plain chips made in the U.S.

Ice-nine.

A.I. elegans.

THEY INVENTED A NEW MOLECULE JUST FOR CRISPS.

ProgrammableWebIfeelgoods Launches API For Rewards Fulfillment

Ifeelgoods, a platform that connects brands with digital rewards, has launched a Rewards Fulfillment API that allows on demand e-gifting. What this means is that brands and apps will be able to automate and streamline the process of rewarding users — a great feature for companies that want to easily create incentives to attract new customers or to keep their current users engaged.

Amazon Web ServicesNew Amazon CloudFront Reporting - Learn More About Your Viewers

My colleague Jarrod Guthrie sent me a blog post with information about four new reports for CloudFront.

-- Jeff;


Amazon CloudFront continues to add reporting features. Recently, launched we've usage charts, cache statistics reports, a popular object report, and near-real time operational metrics via Amazon CloudWatch. Today, CloudFront added four more reports that will give you more visibility into who your end users are - Locations, Browsers, and OS (all grouped under Viewer reports) and a Top Referers report.

Viewer Reports
The Viewer reports include three different ways to look at your users: by the browser that they're using, by the operating system that they're using, and by their geographic location.

Browsers - The Browsers report shows top browsers that your end users used to access your content. You can display either a bar chart or a pie chart by browser name or by browser name and version:

Here's what the pie chart looks like:

The Browser Trends report shows daily trends in requests by browser:

Operating Systems - As with the Browser reports, the Operating Systems report show the top operating systems that your end users are using as well as daily trends by operating system, either as a bar chart or a pie chart:

Locations - You can see the top 50 countries where your end user requests are coming from or the top U.S. states. The report includes the request count from each country, the number of requests in a country as a percent of total requests, and the number of bytes delivered to your end users in each country. You can also display a trends chart that shows how request counts are changing for selected countries or states.

Referrer Reports
Any request for an object on your domain has a HTTP header caller Referer. This header indicates the URL of the webpage from where the request to the object on your domain (or website) was made. The referer header could reference a search engine, another website that links directly to your objects, or your own website. Here's a sample report:

Available Now
The new reports are available now under the Reporting & Analytics section of the CloudFront Management console:

For more information on the CloudFront reporting features, please visit the CloudFront Reports & Analytics page.

If you want to learn even more about these new features, join our upcoming CloudFront Office Hours webinar. on Jan 21st 2015 at 10:00 AM (PT). You can register for this webinar by visiting our Webinars & Videos page and clicking the registration link under the Office Hours Series. Be sure to bookmark this page and check back frequently for new webinars and webinar recordings.

-- Jarrod Guthrie, Product Marketing Manager, Amazon CloudFront

ProgrammableWebDynamsoft Barcode SDK now Supports JavaScript IntelliSense

This article is a company-provided press release and ProgrammableWeb cannot vouch for the accuracy of the statements within. If you have questions regarding the information below, please contact the company that issued the press release.

ProgrammableWeb: APIsInquiry

Open311 allows developers to create third-party apps to report and view citizen neighborhood issues such as potholes to graffiti and more using a city's 311 reporting system. Now on version 1, the RESTful Inquiry API is the tool used to query the 311 database. HTTP GET calls can be sent to the API to list services and expand details on specific services. Information for specific cities accessible with Open311 can be found in the wiki. The API is heavily supported in New York City. The Inquiry API supports XML and JSON formats.
Date Updated: 2014-12-17
Tags: [field_primary_category], [field_secondary_categories]

ProgrammableWebGroove Launches API for Its SaaS Help Desk Solution

Software-as-a-service help desk software provider Groove has launched an API. The JSON REST API allows Groove customers to integrate the company's functionality directly into their apps.

ProgrammableWebSugarCRM Execs Sling Veiled Shot Over API Charges at Salesforce

Though Salesforce wasn’t identified by name, SugarCRM execs warn that the API charges associated with some SaaS providers can sneak up on you as a rising cost (particularly where deep integrations are done).  In a SugarCRM world, you install your own instance of the open source software (on-prem or in the cloud) or you have your own instance hosted by a SugarCRM hoster like SiteGround…and there are no limits on API usage (no maximum number of calls, no maximum # of concurrent calls).

ProgrammableWebHow APIs are Improving Transportation

Presenting at APIdays in Paris earlier this month, Manfred Bortenschlager from 3scale and Eric Thiebaut-George from Carma demonstrated how APIs are creating new transport opportunities and solving urban mobility challenges around the world.

Jeremy Keith (Adactio)The Session trad tune machine

Most pundits call it “the Internet of Things” but there’s another phrase from Andy Huntington that I first heard from Russell Davies: “the Geocities of Things.” I like that.

I’ve never had much exposure to this world of hacking electronics. I remember getting excited about the possibilities at a Brighton BarCamp back in 2008:

I now have my own little arduino kit, a bread board and a lucky bag of LEDs. Alas, know next to nothing about basic electronics so I’m really going to have to brush up on this stuff.

I never did do any brushing up. But that all changed last week.

Seb is doing a new two-day workshop. He doesn’t call it Internet Of Things. He doesn’t call it Geocities Of Things. He calls it Stuff That Talks To The Interwebs, or STTTTI, or ST4I. He needed some guinea pigs to test his workshop material on, so Clearleft volunteered as tribute.

In short, it was great! And this time, I didn’t stop hacking when I got home.

First off, every workshop attendee gets a hand-picked box of goodies to play with and keep: an arduino mega, a wifi shield, sensors, screens, motors, lights, you name it. That’s the hardware side of things. There are also code samples and libraries that Seb has prepared in advance.

Getting ready to workshop with @Seb_ly. Unwrapping some Christmas goodies from Santa @Seb_ly.

Now, remember, I lack even the most basic knowledge of electronics, but after two days of fiddling with this stuff, it started to click.

Blinkenlights. Hello, little fella.

On the first workshop day, we all did the same exercises, connected things up, getting them to talk to the internet, that kind of thing. For the second workshop day, Seb encouraged us to think about what we might each like to build.

I was quite taken with the ability of the piezo buzzer to play rudimentary music. I started to wonder if there was a way to hook it up to The Session and have it play the latest jigs, reels, and hornpipes that have been submitted to the site in ABC notation. A little bit of googling revealed that someone had already taken a stab at writing an ABC parser for arduino. I didn’t end up using that code, but it convinced me that what I was trying to do wasn’t crazy.

So I built a machine that plays Irish traditional music from the internet.

Playing with hardware and software, making things that go beep in the night.

The hardware has a piezo buzzer, an “on” button, an “off” button, a knob for controlling the speed of the tune, and an obligatory LED.

The software has a countdown timer that polls a URL every minute or so. The URL is http://tune.adactio.com/. That in turn uses The Session’s read-only API to grab the latest tune activity and then get the ABC notation for whichever tune is at the top of that list. Then it does some cleaning up—removing some of the more advanced ABC stuff—and outputs a single line of notes to be played. I’m fudging things a bit: the device has the range of a tin whistle, and expects tunes to be in the key of D or G, but seeing as that’s at least 90% of Irish traditional music, it’s good enough.

Whenever there’s a new tune, it plays it. Or you can hit the satisfying “on” button to manually play back the latest tune (and yes, you can hit the equally satisfying “off” button to stop it). Being able to adjust the playback speed with a twiddly knob turns out to be particularly handy if you decide to learn the tune.

I added one more lo-fi modification. I rolled up a piece of paper and placed it over the piezo buzzer to amplify the sound. It works surprisingly well. It’s loud!

Rolling my own speaker cone, quite literally.

I’ll keep tinkering with it. It’s fun. I realise I’m coming to this whole hardware-hacking thing very late, but I get it now: it really does feel similar to that feeling you would get when you first figured out how to make a web page back in the days of Geocities. I’ve built something that’s completely pointless for most people, but has special meaning for me. It’s ugly, and it’s inefficient, but it works. And that’s a great feeling.

(P.S. Seb will be running his workshop again on the 3rd and 4th of February, and there will a limited amount of early-bird tickets available for one hour, between 11am and midday this Thursday. I highly recommend you grab one.)

Matt Webb (Schulze & Webb)Red, yellow, green, bice, plunket, plaid

Opening lines of Wikipedia articles on various colours:

  • Red. Red is the color at the end of the visible spectrum next to orange and at the opposite end from violet. The wavelength of red light is approximately 620–740 nm on the electromagnetic spectrum. Red is the color of blood ...
  • Yellow. Yellow /[phonetic]/ is the color of gold, butter, and ripe lemons. In the spectrum of visible light, and in the traditional color wheel used by painters, yellow is found between green and orange. It is a primary color in subtractive color. Yellow is commonly associated with gold ...
  • Green. Green is a color on the spectrum of visible light, located between blue and yellow. It is evoked by light with a predominant wavelength of roughly 495-570 nm. In the subtractive color system, used in painting and color printing, it is created by a combination of yellow and blue, or yellow and cyan ...
  • Blue. Blue is the colour of the clear sky and the deep sea. It is located between violet and green on the optical spectrum. Surveys in the U.S. and Europe show that blue is the colour most commonly associated with harmony, faithfulness, confidence, distance, infinity, the imagination, cold, and sometimes with sadness.

Do nanometers help?

p215-218: Table 33, Color words in Samuel Johnson's Dictionary (1755).

(I found a file on my computer with the above title. Pages 215-218 of what? The notes are probably from when I was researching Making Senses back in 2006... but the actual source? Possibly Folk Taxonomies in Early English (Anderson). Dunno. Anyway, here are my favourites.)

Black

  • bistre "A colour made of chimney soot boiled, and then diluted with water, used by painters in washing their designs."
  • lutarious "Of the colour of mud."

White

  • grisly Defined as "Dreadful; horrible; hideous; frightful; terrible," but several citations make it clear that grisly is a color word.
  • hoar "1. White. 2. Grey with age. 3. White with frost."

Red

  • lake "A middle colour, betwixt ultramarine and vermilion, yet it is rather sweet than harsh. It is made of cochineal"
  • roan Citation: "Roan horse is a horse of a bay, sorrel, or black colour, with grey or white spots interspersed very thick."
  • rubican "Rubican colour of a horse is one that is bay, sorrel, or black, with a light, gray, or white upon the flanks, but so that this grey or white is not predominant there."

Green

  • bice "The name of a colour using in painting. It is either green or blue."

Blue

  • ceruleous, cerulean "Blue, sky coloured."
  • plunket "A kind of blue colour."
  • welkin "sky-coloured"

Also

Also, hyper-red.

Synaesthesia is when you, for example, "see" the printed number 5 as green, and 2 as green. Or hear C-sharp as blue. I swear I remember reading about an experiment where - when a synaesthesiac sees the number 5 as red - the number 5 is also printed in red. And the resulting colour: HYPER-RED.

But I've been combing The Phenomenology of Synaesthesia (Ramachandran and Hubbard) which is the go-to paper on such questions (for example, Does it matter whether the letters are upper or lower case? Yes it does)... and I can't find anything. Am I mis-remembering?

Finally: A list of fictional colours. Plaid is one of the colors outside of the natural human spectrum visible to large intelligent arachnids in Vernor Vinge's novel A Deepness in the Sky. Cracking book that.

ProgrammableWebHow Open APIs Can Move Healthcare IT Toward Interoperability

Discussion in the healthcare IT world is currently focused on interoperability. The hope is to standardize the exchange of Electronic Health Record (EHR) across systems to enable the instant availability of pertinent health information.

ProgrammableWeb: APIsFT Headline

Financial Times is a business intelligence service that extracts vital information critical to business news. The FT Headline API enables developers to create a newsfeed of relevant real time business news for integration into 3rd party websites or applications. These headlines can be ported into existing CRM, newsletters, and information portals.
Date Updated: 2014-12-16
Tags: [field_primary_category], [field_secondary_categories]

ProgrammableWeb: APIsGoogle Enterprise License Manager

Located within the Admin SDK, the Google Enterprise License Manager API enables enterprise developers to programmatically manage their licenses. The API can be used to revoke a license, return a specific user's licensee by inputted product SKU, create and assign licenses, list all licenses associated with a specific product, and reassign SKUs with licenses. The API is constructed to be processed with HTTP and JSON formats.
Date Updated: 2014-12-16
Tags: [field_primary_category], [field_secondary_categories]

ProgrammableWeb: APIsYes or No

The simple Yes or No API is made for automating boolean decision making within mobile environments. API calls will return a "Yes", "No" or "Maybe." Returns include fun links to comical Gifs, such as Hugh Laurie or Client Eastwood nodding in approval, or Jim from the Office excitedly mouthing "Yes!", or an uncomfortably slow camera zoom on Zach Galifianakis.
Date Updated: 2014-12-16
Tags: [field_primary_category], [field_secondary_categories]

ProgrammableWeb: APIsmigme

migme is a social media service that offers chat, microblogging, and entertainment integrations to increase monetization through marketing and user onboarding. The RESTful migme API allows access to it's 9 million user base, enabling programmatic social media functionality, such as the acquisition of comprehensive user profile information, and the ability to invite friends, make a post, and bill a user for in-app purchases. The migme API is processed using HTTP requests and supports JSON formatting.
Date Updated: 2014-12-16
Tags: [field_primary_category], [field_secondary_categories]

ProgrammableWeb: APIsClearent

Clearent is an integration tool for eCommerce, combining sites, shopping carts, mobile interfaces, payment processing, and Point of Sale applications to create comprehensive payment-enabled eCommerce applications. The API is compatible with shopping carts such as ZenCart, OpenCart, OSCommerce, and custom user applications. Using the Clearent API, providers can process sales, authorizations, captures, refunds, and voids. The RESTful Clearent API is processed via HTTP. All requests require an API key for access. Clearent offers an API sandbox for testing purposes.
Date Updated: 2014-12-16
Tags: [field_primary_category], [field_secondary_categories]

ProgrammableWebHow 200-Year-Old Citibank Totally Nailed Its Hackathon

The Fortune 500 had better take notice. Citi, a 202-year old institution with 250,000 employees in a stodgy industry perceived to be more boring than watching paint dry, didn't just nail its first public hackathon in the U.S. — it may have broken the mold.

ProgrammableWebClearent Launches Developer Center, V2 of its Transaction API

Clearent, one of the nation's fastest-growing payment processing companies, has launched the Clearent Developer Center.

ProgrammableWebOracle Envisions Mobile Driving Multitier Cloud Architectures

Thanks to the proliferation of REST APIs across the enterprise and the adoption of JavaScript on both the client and server sides of applications, Oracle envisions the emergence of multiple tiers of cloud computing services across the enterprise in 2015.

ProgrammableWebW3C&#039;s Battery Status API Powers Web App Usability

The W3C's Battery Status API defines a way for developers to determine the battery status of a hosting device, enabling them to create more power-efficient Web content and applications. The Battery Status API, a candidate recommendation (CR), can be used to throttle back an application when a device's battery is low.

Amazon Web ServicesAWS Week in Review - December 8, 2014

Let's take a quick look at what happened in AWS-land last week:

Monday, December 8
Tuesday, December 9
Wednesday, December 10
Thursday, December 11
Friday, December 12

Here are some of the events that we have on tap for the next week or two (visit the AWS Events page for more):

Stay tuned for next week! In the meantime, follow me on Twitter and subscribe to the RSS feed.

-- Jeff;

Matt Webb (Schulze & Webb)Coffee morning three

Pop the date in your calendar! Coffee morning three is this week. Sort-of-hardware-ish.

Thursday 18 December, 9.30am till whenever, the Book Club in Old St.

Coffee morning two was fun. This will be the same... Zero structure, many conversations all about nonsense maybe with a slight hardware bent, a half dozen or so people, open to anyone!

I've been thinking about why I'm organising this coffee mornings, beyond the whole "there isn't enough time in the day to meet all the interesting people I'd like to, so meet everyone on Thursday mornings" thing, and because I really enjoy introducing people to other people and having that work. I think it's because there's a mode of thinking which I miss now I'm no longer working in a studio, and that's informality. It's the tea-in-the-kitchen chats that make me laugh and spark new thoughts. And that sort of informal serendipity comes from a weird mix of rhythm and randomness. Which means I like having a regular time but not regular attendees. It's just whoever fancies coming that day... I don't want to build a community! But maybe a street corner. I think I'll carry on these coffee mornings into 2015, every couple of weeks probably.

So, next coffee morning is this Thursday, hopefully see you there, and let's chat! If you see someone you don't know, say hello, and if you think two people should talk then make that happen! Recreational catalysis.

There may be crackers containing festive hats. It depends on how organised I am.

Come along!

ProgrammableWebMaximize Holiday Fuel Economy With .Gov Tool

As the holidays approach, many travelers will face high expenses, most notably for drivers consuming fuel on long distance road trips to visit family and friends.

In Digital Gov's weekly briefing on government-supported APIs, Bill Brantley recommends using the FuelEconomy.gov Trip Calculator to maximize your gas mileage. 

ProgrammableWebSaavn Leverages Twitter to Create User-Generated Radio Stations

Twitter’s #Music app was pulled from the App Store in April 2014, just a year after its launch. However, the social network is using a different approach to its most recent venture into the competitive world of music streaming.

ProgrammableWeb: APIsContext Matters

The Context Matters Reimbursement Tracker is a platform that pharmaceutical companies and Data Scientists are using for data analysis. The database is large, with an API in Beta that enables developers to perform analysis more efficiently with integration options and more complexities. The Context matters API allows customers to perform complex database queries and integrate existing systems with the Context Matters data. The API is currently in a Beta mode with access granted in a case by case basis.
Date Updated: 2014-12-15
Tags: [field_primary_category], [field_secondary_categories]

ProgrammableWeb: APIsSmartNotify

The SmartNotify API allows developers to track and analyze communication habits of customers. The aim is to provide optimized integration for communication mediums such as emails, phone calls, text messaging and social media. It also features predictive algorithms for preferred contact methods and analytic tracking.
Date Updated: 2014-12-15
Tags: [field_primary_category], [field_secondary_categories]

ProgrammableWebGoogle Cardboard VR Update Includes Android, Unity SDKs

Virtual reality is an area that is sure to get more mainstream in the years to come. Often it has been associated with expensive hardware and programming complexity. Among all the big-ticket announcements at Google I/O 2014, one item caught everyone's attention: Google Cardboard.

Matt Webb (Schulze & Webb)Filtered for storytelling

1.

Cracking profile of Billy Joel in the New Yorker from October, Thirty-three hit wonder.

Long. Full of good word nuggets.

The saxophone is the radiocarbon.

I seem to be reading a lot of New Yorker articles recently. My current magazine subscriptions are The Economist and the London Review of Books which is max capacity. Maybe time to change it up.

2.

There's a new place in Shoreditch that only sells breakfast cereal. It's called Cereal Killer Cafe. There's a portrait of Hannibal Lector made out of Cheerios on the wall.

Rob Manuel visited, expecting to hate it, and didn't. Lovely story, good luck to them.

3.

I've been totally immersed this weekend in the iPhone game A Dark Room -- minimalist, just text and tapping, and what a picture it paints.

Don't read any reviews, just play it with no preconceptions. Absolutely top fucking notch, best game I've played all year.

Once you have played, here's the development blog.

Also on my iPhone:

  • Workflow which lets you link together actions ("take a photo", "make a GIF", "tweet it") and create custom, automated workflows that you can run from the home screen or wherever there's a Share button. Official site. The app has tons of examples.
  • Pocket Storm plays a thunderstorm from crickets to deluge to fading out again, you choose how long it lasts from 5 minutes to an hour. Different each time.

4.

Very excited -- Adam Curtis has a new film out in January: Bitter Lake.

Politicians used to have the confidence to tell us stories that made sense of the chaos of world events. But now there are no big stories and politicians react randomly to every new crisis - leaving us bewildered and disorientated. And journalism - that used to tell a grand, unfurling narrative - now also just relays disjointed and often wildly contradictory fragments of information.

Here's the trailer. (Down at the bottom of the blog post.) So good.

Curtis' style is distinctive -- a collage of archive footage and music with CAPS stamped over it, and the essay in his own voice. This new film is about the stories that politicians tell - and Afghanistan and all the usual politics - but also looks like it'll be about journalism and his own technique:

It tells a big story about why the stories we are told today have stopped making sense. But it is also an experiment in a new way of reporting the world. To do this I’ve used techniques that you wouldn’t normally associate with TV journalism. My aim is to make something more emotional and involving - so it reconnects and feels more real.

Looking forward to this enormously.

Curtis' The Century of the Self (2002) is on Vimeo -- part 1 here.

Books

I recently finished The Art of Captaincy by Mike Brearley. Brearley was England cricket captain in the late 1970s, and one of the most successful in recent decades. Then later, President of the British Psychoanalytical Society. The book is exactly as excellent as you can imagine -- and has a tendency to illustrate points with detailed anecdotes about moisture on the wicket and fielding positions.

And also The Cyberiad by Stanislav Lem, funny short stories about robots who invent weird things in a galactic civilisation of robots. Here's how it opens: One day Trurl the constructor put together a machine that could create anything starting with n. -- read How the World Was Saved.

ProgrammableWebGulp Launcher Automates Install of Node.js Toolchains

James Ward has teamed up with long-time collaborator Bruce Eckel to tackle the issues facing developers in setting up the gulp toolchain as part of Node.js. With lengthy installation times (especially on Windows) and straightforward documentation difficult to come by, the pair created gulp launcher, which can run gulp with only one download and one command:

ProgrammableWebSource Voice-Over Talent With On-Demand Voice.com API

This article is a company-provided press release and ProgrammableWeb cannot vouch for the accuracy of the statements within. If you have questions regarding the information below, please contact the company that issued the press release.

ProgrammableWebFord Sync 3 Trades in Microsoft for Blackberry QNX

Ford this week revealed Sync 3, a complete rewrite of its in-car, voice-controlled smartphone connectivity system. Sync 3 is better in nearly every way for end users, but Ford made some huge and eyebrow-raising changes to the platform under the hood that will impact developers, too. Here's the scoop. 

Bob DuCharme (Innodata Isogen)Hadoop

What it is and how people use it: my own summary.

Hadoop logo

The web offers plenty of introductions to what Hadoop is about. After reading up on it and trying it out a bit, I wanted to see if I could sum up what I see as the main points as concisely as possible. Corrections welcome.

Hadoop is an open source Apache project consisting of several modules. The key ones are the Hadoop Distributed File System (whose acronym is trademarked, apparently) and MapReduce. The HDFS lets you distribute storage across multiple systems and MapReduce lets you distribute processing across multiple systems by performing your "Map" logic on the distributed nodes and then the "Reduce" logic to gather up the results of the map processes on the master node that's driving it all.

This ability to spread out storage and processing makes it easier to do large-scale processing without requiring large-scale hardware. You can spread the processing across whatever boxes you have lying around or across virtual machines on a cloud platform that you spin up for only as long as you need them. This ability to inexpensively scale up has made Hadoop one of the most popular technologies associated with the buzzphrase "Big Data."

Writing Hadoop applications

Hardcore Hadoop usage often means writing the map and reduce tasks in Java programs that must import special Hadoop libraries and play by Hadoop rules; see the source of the Apache Hadoop Wiki's Word Count program for an example. (Word count programs are ubiquitous in Hadoop primers.) Then, once you've started up the Hadoop background processes, you can use Hadoop command line utilities to indicate the JAR file with your map and reduce logic and where on the HDFS to look for input and to put output. While your program runs, you can check on its progress with web interfaces to the various background processes.

Instead of coding and compiling your own JAR file, one nice option is to use the hadoop-streaming-*.jar one that comes with the Hadoop distribution to hand off the processing to scripts you've written in just about any language that can read from standard input and write to standard output. There's no need for these scripts to import any special Hadoop libraries. I found it very easy to go through Michael G. Noll's Writing an Hadoop MapReduce Program in Python tutorial (creating yet another word count program) after first doing his Running Hadoop on Ubuntu Linux (Single-Node Cluster) tutorial to set up a small Hadoop environment. (If you try one of the many Hadoop tutorials you can find on the web, make sure to run the same version of Hadoop that the tutorial's author did. The 2.* Hadoop releases are different enough from the 1.* ones that if you try to set up a distributed file system and share processing across it using a recent release while following instructions written using a 1.* release, there are more opportunities for problems. I had good luck with Hardik Pandya's "How to Set Up a Multi-Node Hadoop Cluster on Amazon EC2," split into Part 1 and Part 2, when I used the same release that he did.)

Hadoop's native scripting environments

Instead of writing your own applications, you can take advantage of the increasing number of native Hadoop scripting languages that shield you from the lower-level parts. Several popular ones build on HCatalog, a layer built on top of the HDFS. As the Hortonworks Hadoop tutorial Hello World! – An introduction to Hadoop with Hive and Pig puts it, "The function of HCatalog is to hold location and metadata about the data in a Hadoop cluster. This allows scripts and MapReduce jobs to be decoupled from data location and metadata like the schema. Additionally since HCatalog supports many tools, like Hive and Pig, the location and metadata can be shared between tools." You can work with HCatalog directly, but it's more common to use these other tools that are built on top of it, and you'll often see HCatalog mentioned in discussions of those tools. (For example, the same tutorial refers to the need to register a file with HCatalog before Hive or Pig can use it.)

Apache Hive, according to its home page, "facilitates querying and managing large datasets residing in distributed storage. Hive provides a mechanism to project structure onto this data and query the data using a SQL-like language called HiveQL." You can start up Hive and enter HiveQL commands at its prompt or you can pass it scripts instead of using it interactively. If you know the basics of SQL, you'll be off and running pretty quickly. The 4:33 video Demonstration of Apache Hive by Rob Kerr gives a nice short introduction to writing and running Hive scripts.

Apache Pig is another Hadoop utility that takes advantage of HCatalog. The "Pig Latin" scripting language is less SQL-like (but straightforward enough) and lets you create data structures on the fly so that you can pipeline data through a series of steps. You can run its commands interactively at its grunt shell or in batch mode from the operating system command line.

When should you use Hive and when should you use Pig? It's a common topic of discussion; a Google search for "pig vs. hive" gets over 2,000 hits. Sometimes it's just a matter of convention at a particular shop. The stackoverflow thread Difference between Pig and Hive? Why have both? has some good points as well as pointers to more detailed discussions, including a Yahoo developer network discussion that doesn't mention Hive by name but has a good description of the basics of Pig and how it compares to an SQL approach.

You know what would be cool? A Hive adapter for D2R.

Hive and Pig are both very big in the Hadoop world, but plenty of other such tools are coming along. The home page of Apache Storm tells us that it "makes it easy to reliably process unbounded streams of data, doing for realtime processing what Hadoop did for batch processing." Apache Spark provides Java, Scala, and Python APIs and promises greater speed and an ability to layer on top of many different classes of data sources as its main advantages. There are other tools, but I mention these two because according to the recent O'Reilly 2014 Data Science Salary Survey, "Storm and Spark users earn the highest median salary" of all the data science tools they surveyed. Neither is restricted to use with Hadoop, but the big players described below advertise support for one or both as advantages of their Hadoop distributions.

Another popular tool in the Hadoop ecosystem is Apache HBase, the most well-known of the column-oriented NoSQL databases. It can sit on top of HDFS, and its tables can host both input and output for MapReduce jobs.

The big players

The companies Cloudera, HortonWorks, and MapR have gotten famous and made plenty of money selling and supporting packaged Hadoop distributions that include additional tools to make them easier to set up and use than the Apache downloads. After hearing that HortonWorks stayed closer to the open source philosophy than the others, I tried their distribution and found that it includes many additional web-based tools to shield you from the command line. For example, it lets you enter Hive and Pig Latin commands into IDE-ish windows designed around these tools, and it includes a graphical drag-and-drop file browser interface to the HDFS. I found the tutorials in the "Hello World" section of their Tutorials page to be very helpful. I have no experience with the other two companies, but a Google search on cloudera hortonworks mapr finds a lot of discussions out there comparing the three.

Pre-existing big IT names such as IBM and Microsoft have also jumped into the Hadoop market; when you do a Google search for just hadoop, it's interesting to see which companies have paid relatively how much for Google AdWord placement.

Hadoop's future

One of Hadoop's main uses so far has been to batch process large amounts of data (usually data that fits into one giant table, such as server or transaction logs) to harvest summary data that can be handed off to analytics packages. This is why SAS and Pentaho, who do not have their own Hadoop distributions, have paid for good Google AdWord placement when you search for "hadoop"—they want you to use their products for the analytics part.

A hot area of growth seems to be the promise of using Hadoop for more real-time processing, which is driving the escalation in Storm and Spark's popularity. Even in batch processing, there are still plenty of new opportunities in the Hadoop world as people adapt more kinds of data for use with the growing tool set. The "one giant table" representation is usually necessary to ease the splitting up of your data for distribution across multiple nodes; with my RDF hat on, I think there are some interesting possibilities for representing complex data structures in Hadoop using the N-Triples RDF syntax, which will still look like one giant three- (or four-) column table to Hadoop.

Cloudera's Paolo Castagna has done some work in this direction, as described in his presentation "Handling RDF data with tools from the Hadoop ecosystem" (pdf). A more recent presentation Quadrupling your Elephants: RDF and the Hadoop Ecosystem by YarcData's Rob Vesse shows some interesting work as well, including the beginnings of some Jena-based tools for processing RDF with Hadoop. There has been some work at the University of Freiberg on SPARQL query processing using Hadoop (pdf), and SPARQL City also offers a SPARQL front end to Hadoop-based storage. (If anyone's looking for a semantic web project idea, you know what would be cool? A Hive adapter for D2R.) I think there's a very bright future for the cross-pollination for all of these tools.


Please add any comments to this Google+ post.

ProgrammableWebCloud Elements Releases CRM &amp; Marketing Automation API &quot;Hub&quot;

Cloud Elements, a “cloud API integration and aggregation service” for developers, aims to be an abstraction layer-like hub for developers wanting to build apps that integrate with popular Software-as-a-Service (SaaS).

This week Cloud Elements released CRM and Marketing Hubs aimed at making it easy for developers to integrate their apps with multiple SaaS products at one time.

ProgrammableWebNew Voices.com API Exposes Voice-Over Database

Voices.com, an industry-leading website that connects businesses with professional voice talent, has released an API that provides access to its database of more than 125,000 voice-over professionals. The API allows businesses to integrate on-demand Voices.com talent directly into a corporate website.

ProgrammableWebNew U.S. DOJ APIs Expose Legal Jobs and News

This week, the Department of Justice launched two new APIs: the DOJ News API and the DOJ Law Jobs API.

ProgrammableWebNew BlueSnap Payment APIs Enhance Holiday Shopping

This article is a company-provided press release and ProgrammableWeb cannot vouch for the accuracy of the statements within. If you have questions regarding the information below, please contact the company that issued the press release.

Shelley Powers (Burningbird)Shrewsbury: Landlord Retaliation in the Land of Ferguson

photo of smoker

Received this letter yesterday:

Georgetown Apartments will not be renewing your lease, which expires on February 28, 2015. You are still responsible for rent and utilities through the end of your lease agreement.

We are asking you please vacate your apartment by February 28, 2015 to avoid eviction procedures. There is no exception you must vacate the apartment by February 28, 2015.

Sincerely,

Katie Scott
Manager

And so ends our 14 year tenure at Georgetown Apartments in Shrewsbury, Missouri. Shrewsbury is another of the 62 inner loop towns that plague the St. Louis area.

We pay our rent on time, take care of our place, even planted pots with sunflowers that the neighbors loved. We decorate for Christmas, respect people's property, don't have wild parties, and keep the sound down after 10. In fact we've never had problems, until Mark moved in downstairs two years ago.

To say that Mark is the neighbor from hell is to put it mildly. He hit my car with his moving van, he spends over $400 a month on cigarettes (and we live in an older, wood frame building), and, as his ex-wife let slip when she was helping him move, the home he lived in three years ago burned to the ground. His son currently living with him was just released from jail for the second time—this time was for stealing a car, the previous time (in 2013) was for breaking and entering and stealing a gun.

We managed, though. We bought five 300 square foot air filters to help with the cigarette smoke, and purchased specialized carbon-filters for the furnace air vent. We'd stop by to chat with Mark when he was out on the front porch when we went by. I would commiserate with Mark when he talked about his son and his son's various escapades. All was manageable, until Mark got both a smoker-barbeque and a full size, double wide smoker—the kind where you start a fire in wood chips at the bottom, and then the smoke vents out the top. Well, that smoke venting was right underneath our deck, and directly into our home. The first time he ran that thing, our home filled with smoke, leaving us coughing and trying to air the place out from the front, except the smoke curled around the building and came in the front, too. Our home reeked of smoke for weeks.

I complained to the apartment, and the assistant manager basically said tough luck, that there are no rules against barbecues. I said this wasn't a barbecue. She said Mark told them it was a barbecue-smoker. I said, well, he had one of those, too, but this was a smoker. A full size, double wide meat smoker.

I told her about the layer of smoke that surrounded the block, our place being full of smoke, invited them to come see for themselves. Too bad, so sad, and she'd made her decision. I appealed to the manager, who said the same thing.

OK, not much left to do except appeal to the county health board for an air pollution violation, and the local Shrewsbury Fire Department for potential fire hazard. You see the smoker and the barbecue both used 20+ pound LP tanks, and if you look straight down from my deck, you'll see the ground around both littered with cigarette butts. Now, exactly why did Mark's previous home burn to the ground?

The Fire Chief responded with an interesting piece of news: Shrewsbury has fire laws that prohibit barbecues and LP tanks within 10 feet of combustible surfaces in multi-family dwellings. Point of fact, most communities now have these laws in place, though few actually enforce them.

Katie's reaction was a variation of "everyone is going to move because we can't have barbeque grills".

Please understand if we were to ban barbeque grills that would affect 95% of our residents, the majority of the property will be upset and move out which would result in rents increasing substantially. The office has not received any other complaints concerning the smoker or barbecue grills. I am not sure if you and your neighbor below you get along however; we have addressed all of your complaints previously.

Actually, telling us "tough shit" on that smoke pouring into our home really isn't addressing a complaint. And if you want to keep people from moving, put washer/dryers in every unit. And stop being an asshole.

Mark then proceeded to put his smoker-barbecue into the lawn in front of his place, which evidently the management agreed to. It was an odd agreement, because we've been told to keep our lawns free of obstructions. I complained again to the office, and they told me Mark had cleared it with the fire chief. Unlikely, considering how close it still was to the deck, so I filed a complaint with the Fire Department. They eventually came out, measured the distance and told Mark there is no place in his yard he could place either his smoker or his smoker-barbeque that would be far enough away to meet the safety requirements.

These safety requirements aren't arbitrary. Outdoor barbeques and outdoor smoking are leading causes of apartment fires in the US, exceeded only by indoor cooking fires and smoking. And LP tank grills are responsible for the majority of the fires. Just this summer, someone's barbeque caught on fire on a wooden deck in St. Louis, and a firefighter was burned trying to turn the tank off.

And who would really like to live with a full size smoker directly underneath their deck, with smoke pouring into their homes?

Eventually we stopped the smoker, but the cost was a terminated lease. Unfortunately, Missouri is one of the few states that doesn't have a law against landlord retaliation. If you go to the fire department, county health, or other organization about safety or health concerns about your home, your landlord can retaliate by terminating your lease.

By going to the Fire Department, I committed the ultimate sin: I bucked the manager. I continued to push, I rocked the boat. And the manager would rather have that smoker with the felon son, the trashed front yard, and the assorted health and safety risks, than someone who disputed her control.

I'll have more to say on this, and my experiences with the Shrewsbury fire, police, and town council related to this event in a later post. For now, we're buying a home, and I have a lot of research to accomplish in a very short period of time.

By the way, Merry Christmas.

Oh, if you're looking for a place to live in St. Louis, can I suggest Kansas City, instead?

Matt Webb (Schulze & Webb)We Didn't Start the Fire Pedia

Here's a list of all the things in We Didn't Start the Fire by Billy Joel, ordered by Wikipedia article popularity (page visits in the month of November 2014).

Most popular at the top. Page visits in parentheses.

List of pages mostly taken from the Wikipedia entry for We Didn't Start the Fire, and page view count from the Wikipedia article traffic statistics service.

ProgrammableWebPostmates Launches API for On-Demand Local Delivery

On-demand local delivery service Postmates yesterday unveiled an API that gives developers the ability to integrate delivery functionality into their apps.

ProgrammableWebSalesforce AppExchange Store Builder Aids App Store Creation

Salesforce.com today announced AppExchange Store Builder, a set of tools through which organizations can build their own customizable app stores on Salesforce. Using the same tools that Salesforce used to build AppExchange, organizations can create a similar environment around a select g

ProgrammableWeb: APIsW3C Vibration

The Vibration API is a simple browser-based device API documented by the W3C Device APIs Working Group. The Vibration API enables developers to trigger the vibration mechanism in a hosting device, such as a smartphone. The vibration length and pattern of the vibrations can be customized. The API offers simple tactile feedback, not fine-grained control. The API should not be confused with sending alerts and notifications, as the Notifications API can handle these requests. The Vibration API does not specify wether or not the hosting device contains a vibration mechanism.
Date Updated: 2014-12-12
Tags: [field_primary_category], [field_secondary_categories]

ProgrammableWeb: APIsW3C Battery Status

The Battery Status API, documented by W3C, is a browser-based API that enables web developers to discover the battery life of a hosting device. This enables web application features to be dynamically catered to a device's battery status, improving power-efficiency and user experience. The API dependences include EcmaScript and WebIDL version 2. The Battery Status API was historically packaged within the W3C System Information API but was excluded to promote additional efficiency and easier implementation. Documentation for the API is in the form of a brief written by W3C and representatives from Intel and Google, including code samples and additional reference material.
Date Updated: 2014-12-12
Tags: [field_primary_category], [field_secondary_categories]

Footnotes

Updated: .  Michael(tm) Smith <mike@w3.org>