How to: IndieWeb Syndicate

TL;DR - Use Max Böck's 11ty/Netlify starter template, so that Twitter syndication comes standard.

IndieWeb syndication - why bother? #

How many social media sites have you signed up for that went belly up? Have you joined a site that provided you a service only to have that service's offering change in a way that didn't suit your needs as they rearranged their payment plans or got gobbled up by a bigger company? Things like:

• Delicious
• Path
• Google Reader
• Stumbleupon
• Flickr
• Gimmebar
• Springpad

I have. Trying to get content that I supplied to them back out of that site/service so that I could retain it for future use has been annoying. If the stuff I find interesting or worth curating was contained on my site and then shared out with others from there, it would remove that necessity from my life entirely.

The fact is, the data I got back from those sites and services was a volume of parsing and processing that I was ill equipped to handle anyway. So, aside from the OPML file I have used to go from one RSS Feed Reader to another, that stuff lays dormant in a file somewhere if I have not lost it in the years since.

The IndieWeb wiki has its own explanation for why it is good to publish on your own site and syndicate elsewhere. They refer to that as POSSE, which is too cheeky an acronym for me, but I don't judge others for using the term. Ok, I might judge a little, but I still agree with the underlying concept of owning my own data and letting other sites and services have access to it, at my discretion.

Privacy concerns #

Syndicating content out from my site to other sites is a choice I actively make, so I don't feel wary about GDPR implications, or those that will come from California and Colorado regulations that are impending soon. I believe those regulations are just the beginning of what will eventually become a more widespread mandate. I'm still scrutinizing whether capturing webmentions back to things I write might be problematic. I don't want to have to write a privacy policy or have opt-ins and -outs just so I can write what I want and have people participate in the conversation. So, for now, this is a one-directional effort: I write then share it out by my own consent.

The guts of syndication #

Max Böck wrote up a nice explanation for how he got notes he writes on his 11ty based site (using Netlify) syndicating out to Twitter. If you are interested in doing this, too, start your reading there. I am not going to cover his explanation because he wrote it better than I can. This entry explains what I had to do using his article as my starter.

Because I'm not quite as savvy at this, I hit a snag that I couldn't figure out on my own. Hopefully, sharing that here will help others avoid making the same mistake.

Lambda what now? #

So, 11ty is the static site generator I'm using and Netlify is the platform that publishes things for me that are beyond my current skill level. Max uses those as well. If you want to skip having to go through this part of the process that I had to, you can go straight to using his starter template and get up and running more quickly. That wasn't an option for me since I already had this site up and running. For me, this had to be an add-on, which meant figuring out what parts of his starter template I needed and getting them to adjust to my set up.

I have a wee bit of interest in learning these things anyway, so I'm writing this up to remind myself what it took to accomplish this. Choosing 11ty & Netlify as the solution for serving up my website means that there is no database generating things on the fly. There are a bunch of files that make up what you see on the website--that's it. That's what makes it "static." To do anything fancier, like getting one of those files to appear elsewhere on the web, a program has to do something. That's where Lambda functions come into play. Netlify has an explanation on their site about what they are and what they do.

The truth is, my Javascript skills are not strong enough to have written the necessary function, so thank goodness Max had already taken care of that part!

Making it work #

1. Apply for Twitter developer access. Steps for applying are in another entry.
2. Put Max's deploy-succeeded.js file in a functions directory so that it will get compiled when netlify-lambda works its magic.
3. Search through my package.json file to see what modules I was missing that the new Javascript function required. For me it was:
   • node-fetch
   • dot-env
   • twitter
   • html-entities
   • netlify-lambda
4. Run npm install at the command line in the terminal to get each of those modules into my build once I had identified them. For example:

   npm install node-fetch

   Running that command in the terminal automatically updates the package.json file to include the module, making it available to my project. The alternative is to manually add the modules to the package.json file, then run npm install to obtain all the packages at once. Both ways work.
5. Add my Twitter API access tokens to a .env file¹. This is where I got tripped up. As Max helped me work through what my issue was--thanks a million, Max!--he shared how to expand the error handling so it might point me in the direction of where my trouble was. It did! I learned that I had "Bad Authentication data" which I did not know before. While I didn't quite know how to fix it, at least I had a pointer that reaching Twitter was the point of failure. I hadn't worked with that a .env file type before, so I copied and pasted what Max had shown on his site, but it was the wrong way to format the contents. Once I figured out how it should be formatted, I let Max know what my issue had been. He updated his 11ty starter template to include an .env.sample file to the repository so that people can avoid making the same mistake I did.
6. Update .gitignore to include the .env file because it is a bad idea to have Twitter API keys posted to a repository on Github where anyone can find them and use them.
7. Update netlify.toml
   • Create functions = "lambda" so that Netlify will know to trigger the files I have stored there when deploying my site.
   • Update the command that builds my site from "eleventy" to "trigger," which is what I named it in my package.json file, explained below.

Since I had not worked with Lambda functions before, I followed an example written by James Quick at scotch.io for how they work in Netlify and how to test them. Once I understood the concept, I managed follow along.

James uses Gatsby for his website, so I had to make an adjustment because I use 11ty. Netlify is expecting a command of "eleventy" to build the site. I don't know if you can chain commands in the netlify.toml file. I know by trial and error that you cannot have multiple command lines with different things being called. I know it can be done in package.json, so I made the change there to run eleventy and compile the lambda function in one call. In order for the function to get triggered, I had to add a few more lines to the scripts portion of my package.json file:

// compile the site and trigger the lambda function
"trigger": "eleventy & npm run build:lambda",

// run the lambda function in my local environment for testing
"start:lambda": "netlify-lambda serve lib/functions",

// this is the command shown in the first line (above) that compiles
// the lambda function
"build:lambda": "netlify-lambda build lib/functions"

Conclusion #

In my opinion, this stuff is not easy. It takes a bit of mental taxation for someone who is not writing programs or doesn't know how servers get the job done for showing websites to the masses. Still, it was really rewarding for me to learn it. While I wouldn't be able to write the necessary code from scratch, at least I have an understanding of what is going on in that code.

I continue to be grateful that the web development community has so many open, sharing individuals who will give of their time to help others. That is one of the things that drew me to make web development my career. I couldn't have gotten where I am today without the generosity of these folks.

I'm also very glad that the IndieWeb movement has progressed far enough to make it easier for people of my skill level to join in the fun. I don't think we are quite yet at the place where non-professionals can get up and running completely, but it feels like it is getting closer.

---

Update: As of some time in mid-April 2019, something broke the Function and it stopped working. In fact, it prevented my site from building. It was fixed by adding encoding to the package bundle. At the command line, typing the following got things working again.

npm install encoding

---

1. Max mentioned that it is possible to store API tokens in Netlify's build environment variables (Settings > Continuous Deployment > Build environment variables) instead of using a .env file. Good to know!