Mostly Bugless

World Data Visualization Prize entry

2023-01-31T00:00:00Z

World Data Visualization Prize

Timing, as they say, is everything. Just as I dusted off my website (not a moment too soon, I assure you), I stumbled upon theWorld Data Visualization Prize.

Now, considering I had less than two weeks to the deadline, a full-time job, and the monumental task of keeping my offspring alive (kudos to my wife, the real MVP here, for holding down the fort), I wasn't exactly betting on a win. But getting onto the longlist and indulging some of my long-neglected ideas? More than satisfying.

The Big Idea: Predicting the Unpredictable

I maintain a lengthy backlog of ideas, oddball thoughts, and existential scares. Writing these down is essentially my brain’s delete button. One recurring theme? Predicting the future, or more precisely - visualizing it.

In a world teeming with technological revolutions, shifting geopolitical landscapes, and the occasional black-swan event (*cough* covid *cough*), trying to foresee the future might seem, well, ludicrous. But certain things, like demographic trends and CO2 levels, aren’t going to change overnight. And fortunately or not, that AI/Robotics genie isn't going back in the bottle anytime soon.

Experts have been pondering these topics for years. So, while predicting the year 2100 might be a bit of a reach, summarizing the next 20-30 years seems like a fair shot. And let's be real, my real motivation? Understanding just how dicey my son's future might be, and pondering the hard questions he's bound to throw my way when he's a teenager.

And guess what? One of the competition categories was pretty much dedicated to this very topic.

Future of Frontiers

I've posted my visualization on this site. It's not a technological marvel by any stretch (do yourself a favor and avoid peeking at the code — I committed some unspeakable programming sins to meet the deadline), but it did accomplish a few key things:

Satisfied my itch to test OpenAI ChatGPT APIs in a practical setting (a topic worth its own blog post)
Visualized expert predictions contextualized by age, giving me the personalized forecast I wanted
Dabbled in fediverse integration, though, in hindsight, this was more of a distraction than anything else.
Reminded me why I steer clear of front-end work for years — it still doesn’t play nice with Firefox (seriously, I was just trying to make a table!)

Seeing what the experts at WGS think about the future was fun, scary, and deeply interesting. Give it a spin via the link below, or check out the recorded GIFs in this article for a quick overview. (Note: not optimized for mobile and doesn’t work on Firefox)

Snapshots

Contextualize predictions with your (or anybody else's) age

Browse and explore future

Search for specific topics

Learn more, find more connections

Get smart answers to your questions using LLM-supported chat

⚠️ The chat is already disabled to control costs.

$ Following links are used to showcase webmentions functionality

Hiring AI

Smart City

Triple Sector Democracy

Hello (Again) World

2023-01-31T00:00:00Z

😁

After a lengthy hiatus, I have decided to bring my blog back to life.

In its original form, I crafted a custom blogging engine from scratch using F#. It was a thrilling side project and deliberately overengineered solution. However as life and professional commitments took precedence, the project lost its place in my list of priorities. What ultimately sealed the fate of my initial site was the excessive hosting costs. I kept the domain name, as I was attached to it already, and seems I was right to expect I will come back to write here once again.

Now, I've chosen to build a static site with Eleventy. Diving back into front-end/full-stack development after years spent focusing on data platforms has been both nostalgic and entertaining. But I have to admit - first and foremost frustrating. Surprisingly little has changed - frameworks evolved, some new took the web dev community by storm. But javascript is still a horrible language, project setup and framework choice are experiences that probably costed me few milimetrs of my remaining hairline. And don't get me started about styling and browsers.

With that said, please bear with me as there are still several issues in the new version, particularly in the CSS department.

If you're interested in subscribing to updates, I'm experimenting with a more personal approach than traditional mailing list managers. Drop me an email, and I'll periodically share new posts and other updates with you - or just use the RSS feed.

The softer side of IT - public speaking

2019-09-05T00:00:00Z

The case for public speaking

This year, I’ve run workshops, seminars, and participated in more presentations than a PowerPoint at a business convention.

I have a secret, though. No one believes it at first. Not only am I an introvert, but my presentation skills have all the natural talent of a brick.

Learning public speaking was one of the best decisions I made early in my career, and today I hope to convince you to do the same. But first, let’s clear one thing up:

Public speaking is not normal

You’ve probably heard that people fear public speaking more than death. Poor science reporting and incompetent journalism aside, there’s a kernel of truth in there worth exploring. If your ancestor was being stared down by the whole tribe, their body would have kicked into overdrive — because either a saber-toothed tiger was behind them or they were about to get thrown out for solitary doom. Neither of which bode well for a future beyond next Tuesday.

Let's be clear: being afraid of public speaking is absolutely normal. If you’re not, you either have the experience of a seasoned pro or somehow throughout schools, family gatherings and work meetings you kept the blissful ignorance of someone who's never had to do it.

Public speaking matters

For those of us who find solace in the glow of a dual-monitor setup, working on soft skills can be about as appealing as a root canal. Why spend time practicing public speaking when you could be optimizing your gnarly Spark performance bottleneck or setting up a PI-cluster for distributed machine learning?

Well, because unless you're the very last IT professional on Earth, you need to communicate. Think about it:

Convince stakeholders vs. bore them into a coma.
Inspire your teammates vs. make them regret ever logging on.
Excite your group about upcoming changes vs send them straight to LinkedIn.

Your brilliant ideas are only as good as your ability to convey them. This is not just idle chit-chat; it's a fundamental career skill.

Don't believe me? Listen to this gem from Warren Buffett, who ranks communication training as his most valuable degree:

They gave us this book of speeches—keynote speech, election speech, lieutenant governor's speech—and we were supposed to deliver these things every week. The way it works is that you learn to get out of yourself. I mean, why should you be able to talk alone with somebody five minutes before and then freeze in front of a group? So they teach you the psychological tricks to overcome this. Some of it is just practice—just doing it and practicing. We really helped each other through. And it worked. That's the most important degree that I have.
Warren Buffett in The Snowball: Warren Buffett and the Business of Life

Warren Buffett, a man who could buy small countries with his pocket change, claims that communication skills can increase your personal value by 50%. And who am I to argue with the Oracle of Omaha?

It's just a skill

Imagine deciding to learn the saxophone. If your first attempt was in front of a packed Wembley Stadium, it would likely be memorable for all the wrong reasons.

Neither the audience, you, nor your unfortunate manager would be thrilled.

Although if you spent 30 hours learning, you could probably perform a simple tune for your friends. Start filling your skill bar — you want it maxed out before any career-defining performances.

Opportunities for public speaking, whether in front of large audiences or during tough small-group meetings, often coincide with crucial career moments. Interviews, pitches, team addresses, and tech talks are all opportunities to shine — or bomb. If you had one shot, one opportunity, would you capture it, or just let it slip? _{Sorry, I couldn't resist}

So how do you make sure you're ready when the spotlight hits?

Start small. Give presentations to your team and solicit feedback. Or check out your local Toastmasters club. Sign up for a class. Record yourself prepping for that tech talk and practice* like your dream job depends on it — because it just might.

Yes, some folks are naturally gifted. Extroverts might have a smoother ride since they practice early, but it doesn’t matter. If you’re willing to put in the hours, you can learn anything. I recorded my first presentation when I started actively learning public speaking. Two years later, watching it was just painful. Cringing hard, I hit ‘delete’ with no regrets.

Buffett was right. It’s one of the most vital skills you can develop.

[2024 Addendum] Five years later, I can definitively confirm and double down on everything I wrote here. I attribute much of my career success to the ability to communicate effectively, and presenting in front of large and/or demanding audiences is a skill that I use almost daily.

* There is some nuance to how you practice effectively. That’s why my core advice is to join a club or find a mentor as a first step. For instance, repeating your entire presentation from beginning to end over and over is likely to not only waste your time but also make your delivery worse.

The softer side of IT - effective learning

2019-06-18T00:00:00Z

Effective learning

In the ever-changing IT ocean, you must keep swimming to stay in your current place. Frameworks change, practices develop, and paradigms shift - a good developer is a learning machine, no matter if they are working with machine learning or javascript.

I'd have to caveat that with few words on unchanging principles and that most of the things we are discussing as novelty have been done 40 years ago, but that's worth another post. But even if you optimize for the right knowledge and avoid trying to remember API details of a tool that will be gone in two years - there is still enough of both constant and constantly changing but essential topics that will last you a lifetime of learning.

And thus, one of the key issues I have with our education system, especially the academia (and as far as I can tell, this is not an isolated issue here), is that they absolutely fail to teach about softer aspects of technical roles - even effective learning.

Spaced repetition

Spaced repetition is, as far as I could find, the best system we have for retaining knowledge. And one that's quite well researched. Long story short, if you are tested right after you start forgetting, the rate at which your memories decay will slow down. So from forgetting 60% of the content in 1 hour

You can go to

and

Finally managing to retain whatever nugget of wisdom you were trying to memorize.

The fact that you need to learn how to learn is a pretty Big Thing. We are what we remember - you can't be a mathematician without remembering the basic formulas. Now a trick question: do you remember all points of OWASP Top 10? Of software craftsmanship manifesto? Or even the Agile Manifesto? In contrast to what happened with manual multiplication and calculators, We can't outsource things that are supposed to drive our behavior.

I'm not going to waste mine or your time to scratch up a good introduction to spaced repetition, while a fantastic 20-minute starter like this already exists, and you can find further details on making great flash cards here.

However, let's explore today the issue of turning this theoretical approach into material results. And let's face it, consistent 10-30 minutes per day of intensive, inconvenient practice is an impossible plan for many, if not most people - me included.

Willpower

I heard about spaced repetition and started using Anki years ago. I'm not going to lie - even though the promise of gaining a severe edge in the knowledge economy is alluring, I've been consistently failing to follow up on regular repetition (or even adding new cards to the set).

Willpower is a funny thing.

In the meantime, I tried multiple habit-building techniques or apps, from 'best in productivity', through notebooks, reminders, and wall-mounted scorecards, to social commitments.

Yes, it did have ~some~ results. No, I'm not happy with them.

Tiny habits

A few months ago, I stumbled upon a Tiny habits program by BJ Fogg. The basic idea about Tiny Habits is that the behaviors you try to implement are:

done at least once per day
take < 30 seconds
require little effort

And are triggered by your existing automatic actions (brushing teeth, setting an alarm clock, opening a fridge, etc.)

In short - habit-forming done right. There are many important details, including making the process fun and focusing on experimentation, but once again, there is no point in me taking the stage. Just sign up for a free 5-day course and let yourself be amazed.

Unexpected synergy

Tiny Habits are currently my number 1 recommendation for making lasting life changes. However, making a habit of daily spaced repetitions is a whole another level of improvement. Hopefully, you will find this combination of a memorization technique with a habit-building system as useful as I and my significant other did!

For reference, I'm using Anki + Anki Droid , with most of the flashcards added on desktop and repetitions done on mobile each morning and evening.

If you are interested in soft skills topics, take a look on my previous post Overview of time management and in any case feel free to comment

Links summary

Mental models in software development

2018-12-31T00:00:00Z

Software development is an amazingly complex enterprise. On the one hand, we have the inherent complexity of getting pieces of silicon to do our bidding and 50 levels of hardware, networks, systems, and frameworks to coordinate. On the other - humans. And those come with relationships, ambitions, visions, and fears.

To get things done reliably, as an industry, we came up with a host of patterns and ideas, big and small. DRY, KISS, Clean Code, Clean Architecture, Design Patterns, TDD, Agile, Software Craftsmanship... and so on and so forth. There is, however, one useful perspective that's even one level higher - let me introduce you to Mental Models.

A mental model is simply a representation of how something works. We cannot keep all of the details of the world in our brains, so we use models to simplify the complex into understandable and organizable chunks.

FarnamStreet

At first sight, it doesn't seem to be an extremely revolutionary observation. Yes, we keep a simplified version of reality in our minds; tell me something I didn't know.

I find the lens of mental models to be an extremely handy tool. The overall framework, as defined on Farnam Street (which you can see linked around my previous posts quite often), seems obvious - find the big cross-domain concepts, understand them and use them daily. The catch comes in work required since this is pretty much a life-long process.

So why even bother? I feel we tend to get overfocused and lost in specific topics or trending approaches in IT. When discussing microservices, DevOps, agile, or SCRUM, you can feel the peer pressure and FOMO. Far too many teams follow along and drive their agenda with hot blog posts or the most recent Gartner materials.

Looking at this from the perspective of mental models is a chance to take a step back and evaluate whatever we are doing from one level higher - hopefully noticing the forest surrounding the current tree in front of us. And as a side bonus, you have a well maintained toolbox of cross-domain tools to apply when needed.

Few relevant examples that stand out for me:

TDD, A/B testing, monitoring, code reviews, and retrospectives - all fall under the general model of Feedback. Looking at those from the perspective of the feedback loop and control engineering principles around it, what would you do differently? Is the feedback loop working for you? Is the information timely and accurate?
Second Order Effects - a lot of the things we do can have positive first-order effects (we shipped without postponing a deadline) and negative or deadly further consequences (code became a mess, and we are now moving at a snail's pace). Having this model embedded in your thought process makes you more likely to notice the problem before it materializes and gives you plenty of ammunition to push back or at least plan mitigation of negative consequences down the line.
- I've seen it argued that it's just sensible analysis and planning, but I find that having a common language and model to use in tradeoff discussion is extremely important. Asking explicitly "What will be next order effects of this decision?" usually yields different answers than "What will be the consequences?"
Antifragility - is the system we are building fragile (fails with little push), robust (can tolerate a lot), or does our process make it better with each failure (antifragile)? How can we use the practices we are considering to get closer to antifragility?

The cited FS guide is an excellent starting point with a broad overview of 109 mental models split into different areas (e.g., Human Nature, Physics, and Numeracy).

Check it out here

Time Management Knowledge Pill

2018-09-14T00:00:00Z

This post is migrated from v1 of this site - there might be couple issues left around formatting or links. Since 2018 also my understanding of time management has become much more nuanced and I would either caveat or outright disagree with some of the content. Consider yourself warned.

Reactions to time management/self-improvement topics vary, but especially among the more skeptical cultures (like Poland) and communities (programmers), you can get some flak just by mentioning them. And I'd dare to say there is some substance behind allergic reactions I have sometimes seen - a lot of the accusations against coaching, Time Management and general self-help industries are not coming from thin air.

Still, unless you are completely satisfied with how your life is going right now, some of the proven techniques and grounded in science ideas can help. At least a little bit. My goal here is not to persuade you to anything, nor promise you life-changing results. However, I hope to show you what I stumbled upon in the last few years and what I'm convinced has at least some merit to it.

Keep in mind that none of the below concepts are deeply explained. There are whole books dedicated to every single one of them, thus I only aim to to gather my thoughts in a roughly sensible manner. And if any of below inspires you to dig deeper, that would be just a bonus.

Core Tenets

There is no such thing as time management

Time moves in one direction and disregarding experiments with sublight velocities no one has ever affected the way it works. What we can manage is ourselves - our actions, attitudes, and emotions. Thinking in terms of managing time as moving blocks of seconds between buckets of activities skips the beautiful complexity of our messy brains and their evolutionary legacy.

Another critical point here is that time is the ultimate constant - everyone, you and me, have the same 24 hours of a day. While the external forces and their personal history make each person's challenges unique, the allocation of those 24 short hours is the only control we have - and the only way to make a difference.

100 blocks a day 52 weeks a year

Time management is a skill

And skills take time to acquire.

You can't expect to be good at any of the techniques the first time you try them
Failure is part of learning
Theory without practice is useless
Repetition is not learning (deliberate practice is a separate skill/concept)
Working on more than one substantial change at a time is a guarantee of failure

No matter if you subscribe to Gladwell's estimate of 10000 hours of practice to mastery, or Tim Ferriss's 6 months, one of the key ideas to remember is that time management is a little bit like cooking. If you keep your ego in check, have good sources or mentors and keep experimenting you can quite quickly get to the point of having solid basics.

Main topics

Energy

No amount of methods or programs are going to make a visible impact on your life if any of the physical pillars are off:

Sleep

One of the most annoying scientific facts is that you need 8 hours of sleep. Sleeping only 6 hours per day in the long term leads to a permanent decrease in performance equivalent to not sleeping for 2 previous nights, but - here is the twist - without any awareness that you are effectively sleep-deprived and ineffective. Yes, I’ve personally tested this.

Keep in mind that our bodies love routine and there is no way you can feel fresh if your wake-up time is scattered between 5 am and 11. Also, watch out for blue light, screens and late evening coffee. I use f.lux for years already, but modern devices seem to be getting screen color control as a basic system feature.

On a side note, when you wake up in December and it's still pitch dark outside at 7 a.m. don't expect to be productive in the morning. Hooking up your lights with an automatic dimmer does wonders for your sleep cycle if you can get them to slowly brighten up before your alarm clock and gradually dim before you go to bed. I use a small set of Philips Hue smart bulbs but in a hindsight I would look for a cheaper option - they are great but not on the price/value ratio while there are many new competing products.

Paper on long-term sleep deprivation Matthew Walker, Why We Sleep: Unlocking the Power of Sleep and Dreams

Exercise

Regular exercise is hard. It takes time. It takes motivation. And lots of people look at time management to help them with establishing a fitness routine, as I do, and maybe you. Egg and a chicken.

Exercise not only helps with avoiding death from any of the cheerful consequences of white-collar chair-sitting. It's non-negotiable for getting maximum performance of your brain.

Move.

Eric Hagerman, Spark Scott Adams, How to Fail at Almost Everything and Still Win Big

Food

Diet is a controversial topic. No matter which particular approach you mention, someone will throw a bunch of research papers and hateful words in your direction. But what we eat defines how we feel physically. A right diet that doesn't make you sleepy throughout the day, nor hate yourself and the person who invented tofu, is a long way to having a high-quality life.

Question, learn and experiment.

Michael Greger, How Not To Die Scott Adams, How to Fail at Almost Everything and Still Win Big

Managing energy

Your energy levels are also affected by time of day, environment and people. Are you most effective in the mornings? Is your brain offline after lunch? Do the people you talk to make you happy and motivated? Does the clutter on your desk make you depressed? Do you have enough of the D3 vitamin?

Trying to solve complex problems when your body is focused on digesting is not going to be efficient. Organizing emails when your brain is sparking with power is just a pure waste. Measure, plan and experiment.

Habits

We are creatures of habit, and most of our decisions are automatic. The science behind this topic is solid, and its insights are essential to help tune your life.

Overcoming bad habits

Why do you want to change this habit?
What is the trigger? What is the reward?
Replace the habit with a new one leaving same trigger and same reward.

While trying to remove the trigger rarely works - good luck removing 'feeling stressed' - when working on replacing the routine it's useful to lock yourself out of the bad alternative. For digital habits, there are blocker applications that cut you off from websites, and many routers have this option.

Technically-able can also edit the /etc/hosts file or write their own scripts that blow up your system if you 'accidentally' get lost in 'one more turn' mode. That's my personal approach right now, but I tried the popular blockers before, and they did also work.

Making new habits

Decompose your goal into a smallest possible action - Tidying bedroom -> making a bed -> spread out the bedding
Decide on the trigger - I woke up
Setup a reminder - sign, place, notification (physical is better than digital)
Dependingly on a habit, you may need between 3 weeks and a 2/3s of a year
It's impossible to build more than one habit at a time

As a physical pattern in your brain emerges you will switch to autopilot. And exercising, planning, reading, healthy eating, or whatever floats your boat - will be completely effortless. Having no need to force yourself in any of the routines you always wanted to have sounds good, right? In fact, if you skip for whatever temporary reason, you are going to feel quite weird.

On a side note, willpower is a separate topic. We can train willpower just like a muscle and people use it to do incredible things. But in everyday life habits eat willpower for breakfast.

Keystone habit

Some of the practices have disproportionate effects on overall life quality. Removing a single worst lousy habit can completely change your direction, like getting back 2 hours of time a day that would be allocated to passive social media browsing. Getting a habit of regular exercising makes you feel better and more energetic.

Also, see the Pareto principle (20/80).

Habit chaining

It's easier to add additional loops on top of existing ones.

Charles Duhigg, The Power of Habit

Bad habit or an addiction?

That's a question you can ask your psychologist. Some of the most common bad habits are on the verge of addiction, like excessive Facebook browsing, Netflix binging and drinking 1 liter of coke a day. It's a serious civilizational problem - technology is breaking our brains.

Nicholas Carr, The Shallows

Mindsets

Our mindsets are kind of mental habits - they ways we think, worry and generally talk to ourselves are rarely conscious. We can take control over them though.

If your mind hates you and talks to you in pathological patterns ("You did it again, idiot.", "Why does this happen to me always?" ), there is a need for change. Also, being able to shut down your mental chatter when falling asleep is a superpower.

Bhante Henepola Gunaratana, Mindfulness in Plain English Albert Ellis, A Guide to Rational Living Marilee Adams, Change Your Questions, Change Your Life

Meaning

Whatever your dreams are, start taking them very, very seriously.

Barbara Sher

The first association to time management is probably a to-do list or a calendar. But those are only tactical measures that can make you more efficient in your current efforts. If you are working on wrong things at work, or overall pursuing an unfulfilling career, you need to go back to the strategic layer.

We need a purpose in things we do - no amount of to-do lists is going to compete with a single vision and some determination. Understanding your own values and what you really want to get out of life is really hard and takes deliberate effort. Every action has to have a 'why.' Preferably - a written down one.

Dan Ariely, TED talk: What makes us feel good about our work Stephen R. Covey, The 8th Habit Chris Guillebeau, The Art of Non-Conformity Daniel H. Pink, Drive

Another important aspect is the base limitation of things we can do within the 24 hours of a day. Fun fact: initially, the word priority had no plural form! Choosing 'the one' thing is an art itself.

Greg McKeown, Essentialism

Flow

Flow is a state when you are completely immersed in an activity. When all your worries are gone and 4 hours feel like 14 minutes, that's flow. And that's when we create the best work we can.

To get into the flow, you need a meaningful task that is right on the edge of your skill level.

Mihaly Csikszentmihalyi, Flow

The Eisenhower Method

A core distinction between what's urgent and what is important.

7 Habits of Highly Effective People

Being busy vs being productive

We live in a culture that promotes busyness. People are proud when they work around the clock and are constantly hooked to a stream of news, posts and notifications. No great work was ever created when switching between email, slack and the piece itself.

Long story short, multitasking doesn't work. As well as cramming your schedule to 105%.

American Psychological Association summary of multitasking research Greg McKeown, Essentialism

Feedback and planning

To risk a cliche, if you improve 1% per day for 365 days, you are going to improve by 3778% in a year. 1.01^365 = 37.78

Hundreds of different systems and ideas can help you to be more effective. Some will work for you, some won't, but will for others. Some will, but not today. And some are just bonkers. As long as you keep an attitude of constant experimentation, you can work out a satisfying routine that moves you toward the impact you want to have on the world - however big or small your vision is.

Jordan Peterson, How to set goals the smart way

Current state and Quantification

What are you spending your time on? What makes you feel good? What drains your motivation?

Without understanding where you are now, you can't plot the route to achievement. Consider:

life audit - writing down every single thing you do for 2-3 days; also useful for understanding your energy levels in a day
time logging - toggl enables you to consistently log your productive (and unproductive if you choose to do so) time split into different buckets
automatic logging - some tools, like RT record everything you do on your device / in your browser and report the results

Goals

Goals are necessary, and there is a whole science behind goal creation (also a very profitable industry). The minimum pattern of SMART means that goals should be:

Specific
Measurable
Achievable
Relevant
Time-bound

Objectives and key results (OKR)

OKRs are a state-of-the-art practice for goal-setting, especially prevalent in successful companies:

John Doerr, Measure What Matters

Systems

Goals do not work. Anyone who has made a New Year's resolutions can attest to that. The alternative approach is to create systems that push you in the right direction every day. So, instead of having a goal of writing a book this year, make a system of writing 150 words every day. And decide when, where and how you will be writing them.

Consider systems as goals done well with supporting latticework of habits and environment design.

Goals

Single point of achievement - constant state of pre-success failure
Delayed gratifications
Large and uncertain - cause anxiety

Systems

Everyday progress - feeling good every time you apply the system
Continuous gratification
Small and manageable

Scott Adams, How to Fail at Almost Everything and Still Win Big

Irrationality

Contrary to how economists would like the world to work, people for most purposes are inherently crazy. We fail our goals, skip diets and predictably make horrible decisions. On the bright side, there is a lot of research on human irrationality already done. Understanding central insights in this area helps to prioritize your next summer body over half a liter of a milkshake.

The ~~official~~ Wikipedia's list of biases is really, really long. Consider starting with the books. Some of the obvious-yet-enlightening ones include

Hyperbolic discounting - discounting reward in time, overvaluing short-term gains
Optimism bias/planning fallacy - we are consistently bad at estimating anything.

Dan Ariely, Predictably Irrational Daniel Kahneman, Thinking, Fast and Slow Tools and minicourses on getting your decision making under control, ClearerThinking.org List of cognitive biases

Procrastination

Procrastination is a chronic avoidance of tasks, whether self or externally imposed. It's closely connected to some of our biases (especially Hyperbolic discounting). Long story short, ambitious, uncertain and important goals cause mental resistance. You can break down those problems with good systems that turn massive undertakings into a small, manageable series of steps.

Procrastination is also often a second-degree effect of

lack of vision
overcommitment (lack of prioritization)
deeply ingrained bad habits

If you find yourself procrastinating consider shrinking the task to a smallest possible bit - 'I will open a notebook.' Pomodoros also help. And a small technique suggested by R. Wiseman: making a deal with yourself that you are going to work 'just for a few minutes' instead of making a deal that you will browse Facebook 'just for a few minutes', which we both know just doesn't work.

Richard Wiseman, 59 Seconds

Base tactics

Pomodoros

Breaking your work into 25-minute long high-intensity, full-focus intervals with breaks between.

Pomodoro technique

Time blocking

Building your schedule on large blocks of focused time. 9-11 deep work, 11-12 bureaucracy and emails, meetings 13-15, etc. Ideally planned tasks should correspond to natural cycles of your physical energy.

Hint: never plan more than 80% of your time. Buffers are crucial.

Environment control

There is no such thing as multitasking. Any kind of creative work requires focus, and there is no way you can get into the flow when you get interruptions every 10 minutes.

Turn off notifications and work in a silent environment. Or at least get a reasonable pair of headphones and use white noise to drown out noises made by those pesky humans.

Regular planning

You need to spend time to save time. 30 minutes on Sunday for a week planning session does wonders, as well as a quick daily check on the most important thing to achieve the next day.

Buffers

Since we are naturally horrible at estimating (which also explains a lot of the IT industry), creating large buffers for tasks is a valid technique to avoid failure. Historical data solves this problem also. If a small size post takes me 6 hours, I can expect next one to not deviate too much. However, if I'm estimating a new work - a large post with heavy research - and I come up with 15 hours top-down, it's safe to look at the 30+ hour range.

50 minute mini-course on improving your planning from ClearerThinking.org

To-do lists

We can hold at best 2-3 things in our brains. To not get lost in the sea of bills, calls and life goals it's essential to have a log of things left to do. Creating effective to-do lists is a vast topic. Some pointers:

Tasks should be as small as possible, whole projects are not tasks and therefore not for to-do lists
Visit everyday
Have separate lists based on time - today, this week and future is a common choice

Don't forget to get things done.

Getting things done

Complex, overhyped system enabling you to do wrong things very efficiently. Based on to-do lists. Sarcasm aside, it's really great and lets you take back control over large amounts of tasks in all areas of your life.

David Allen, Getting things done

Ulysses contracts

Making early commitments that prevent your future you from making bad decisions. Uninstalling time-wasting apps, blocking infinite-scrolling websites on your router, inviting friends over so you have to clean up your flat, etc. Creating artificial deadlines and getting accountability partners who reward and punish you also fall into this category.

Checklists and automation

Checklists are the first step to automation. You shouldn't be spending computational cycles of your brain on what to take for next trip, nor which groceries to buy at the beginning of a week. On the other hand, a lot of our tasks can be automated or outsourced - paying bills, doing the dishes, checking your daily set of blogs for new posts.

Main myths

Visualization / self-affirmation

Thinking about your best future successful self does not work - at best, possibly it's even harmful. There may be some merit to thinking about concrete steps of your journey and visualizing yourself doing the work required though.

Richard Wiseman, 59 Seconds

Working more

Overtime and generally pushing yourself over the limits doesn't work besides single occurings with significant rest periods afterward. Life is a marathon, not a sprint.

There is a single trick, 10 step guide, or a blog post that will change your life

Making yourself more productive means fighting against 30, or however old you are, years of reinforcing experience and 7 million years of evolutionary history. Getting better is hard. And easy answers usually are either incomplete or completely wrong.

Minor concepts

Resistance

Procrastination as a physical force of nature.

Steven Pressfield, The War of Art

Essentialism

Stop doing everything, start doing essential things.

Greg McKeown, Essentialism

Pareto principle

20% of work gives you 80% of results

Jordan Peterson, How to set goals the smart way

F*ck it effect

I ate one piece of pie so I might just as well finish whole. Having contingency plans and non-hostile mindset ('what can i do better next time?' instead of 'I'm a failure' helps).

Inbox 0

Keeping your email inbox completely empty by making quick, radical decisions. When poorly applied leads to an explosion of meaningless emails in your team/organization. When applied correctly helps you remain sane and productive.

Barbell strategy

Mediocrity kills, results are on the extremes. Sharp focus with good rest afterward is an order of magnitude more effective than messing about the whole day.

Nassim Taleb, Antifragile

Via negativa

Inversion of planning. What I absolutely must not do if I want to have a fulfilling life.

Nassim Taleb, Antifragile

Antifragility

Improvement in face of failures and stressors. Applies both to systems (chaos engineering) and humans.

Nassim Taleb, Antifragile

Toolset

Minimal version:

Calendar

There is really no magic here, as long as you can plan your schedule it works. It's a little easier to create recurring events and time-blocks on digital calendars, but they are easier to ignore and disconnect physical sensations from the act of planning.

Todo list

Mentioned above.

Try Trello

Or any other app from the hundreds you can find.

Additional

You can also try

Time loggers
Website blockers
Habit builders (I prefer physical ones to digital, but there is obviously an app for that)
Decision journals

Other notes and wrap-up

Besides external links above, some of general references and inspiration for this post include:

7 Habits of Highly Effective People Farnam Street

Also it's worth mentioning countless time-management and related trainings (including those delivered by me over the years), attended courses and lengthy discussion with a whole lot of smart people, including a quite terrific course by Piotrek from produktywni.pl that I attended literally yesterday.

Yes, there is a lot of things you can try, and the amount of information might seem overwheling. No, don't panic. Start small!

Peak Bad Habit

2018-08-05T00:00:00Z

We have spent the past decade under the sign of habit-forming technologies. Scroll down - get a dopamine hit. Your tweet was seen by another 100 people - one more. Binged an entire season of a new series? - oops.

The most significant successes that come to mind - Facebook, Twitter, Instagram, Netflix, Youtube - all base their achievements in creating a habitual interaction with their products. Habits themselves aren't inherently bad, of course. We base our existence on countless automatic patterns. However, is spending hours a day scrolling through neverending (and mostly never-valuable) content good for your health?

(The obvious and already well established answer is no)

Our operating system is easy to hijack. For human 1.0, all you need is a red blob popping up on the notification bar or an auto-play turned on by default.

Habits

All habit-forming requires is a cue, routine, and reward.

As your brain figures out the connection between them, you get locked into an automatic pattern. That worked great for our ancestors, who were more concerned with staying alive than watching cat videos. And not so great for us when thousands of people work daily to get us hooked on their products with the same mechanisms.

The worst and best thing about habits is that we do them automatically. Life would be pretty dull if I had to decide which pocket to put my wallet in consciously. But when opening your laptop triggers a direct pathway to an infinite-scrolling website, we have a tremendous problem.

(You can read more about habits in the excellent "The power of habit" by Charles Duhigg)

Apple and Google to the rescue?

Both Apple and Google recently revealed upcoming features for iOS and Android which aim at helping users manage their interactions with phones. Usage statistics, blocking apps by the time of day, limiting notifications, etc.

While those are far from revolutionary and don't seem to deliver much over 'just turning your goddamn phone off,' they prove there is enough pressure on digital wellbeing that companies which earn on screen time are willing to help users limit it. That's a good sign! Maybe movements like Time Well Spent and a rising tide of articles and books (The Shallows comes to mind) will slowly move us towards a more conscious future?

A low-tech alternative

Time tracking isn't a new idea. Many apps and companies offer precisely that already. I wasn't delighted with the ones I tried before, mainly due to a tiny little issue I have with running borderline spyware on my devices.

So yes, getting statistics of my screen usage from built-in system functionality sounds amazing.

But before that makes its way to my phone, there is a straightforward exercise that I run from time to time which helps me notice unwanted patterns in my life:

Print out a time log for two days
Every 15 minutes, write down what you were doing

After two days:

Be surprised
Ask yourself is this consistent with your self-perception - and what you want to change

It works wonders for me, and a few ~~guinea pigs~~ people I ~~forced~~ kindly motivated to try it - without depending on a tech giant to solve my problems.

There seems to be also a whole movement and a book by Laura Vanderkam ("168 hours") based on radical time tracking, but I haven't yet gotten around to digging deeper into it.

Peak bad habit

I wonder if we will see more news about digital monopolies taking action to limit their habit-forming impact. If the social pendulum swings back from digitized life back to reality, we all would be better off. But in a world where competition for our attention is a zero-sum game for tech giants, there are good reasons to remain skeptical, even in the face of the first good signs.

Still, there is always the personal choice to pull the plug on any service you know you spend too much time on.

He is free who lives as he wishes to live; who is neither subject to compulsion nor to hindrance, nor to force; whose movements to action are not impeded, whose desires attain their purpose, and who does not fall into that which he would avoid.
Discourses by Epictetus

Post-authenticity society

2018-07-29T00:00:00Z

I'm only 3 books into the series, but The Culture by Iain M. Banks is one of the best pieces of sci-fi I recently read. 30 years ago Mr. Banks in one of the books made a casual remark about authenticity in a society where any kind of proof can be faked with technology. Keep in mind that his vision is one of the most optimistic scenarios for technological advancement where artificial intelligence coexists with mere humans in an utopian society.

(...) Anybody could make up anything they wanted; sound, moving pictures, smell, touch . . . there were machines that did just that. You could order them from a store and effectively paint whatever pictures-still or moving-you wanted, and with sufficient time and patience you could make it look as realistic as the real thing, recorded with an ordinary camera. You could simply make up any film sequence you wanted.

Some people used such machines just for fun or revenge, making up stories where appalling or just funny things happened to their enemies or their friends. Where nothing could be authenticated, blackmail became both pointless and impossible; in a society like the Culture, where next to nothing was forbidden, and both money and individual power had virtually ceased to exist, it was doubly irrelevant.

The funny thing is, we are kind of getting there.

You have most likely heard about 'deep fakes' before. The term blew up in the news when everyone started using machine learning to substitute faces in videos. Political satire, revenge pornography, and Nicolas Cage as the main character of every movie - humans applying ~~artificial intelligence~~ any new technology at its finest.

At the same time, there are startups that aim to perfect voice imitation. Lyrebird lets you create a model of your own voice. They made the news with fake samples for Trump, Clinton, and Obama early last year. Their demo is already taken down, but you can hear the samples on a TechCrunch article. Far from perfect, but a significant step.

It was always possible to alter images, as the classic example above proves. Photoshop just made it much more accessible. The same with videos - with enough resources you can fake any kind of evidence.

We are however far from an utopia where money and power do not matter, like the society of Culture in Bank's fiction. So what can we trust today? And who can we trust tomorrow?

Pumping up your productivity with..noise?

2018-07-22T00:00:00Z

Noising up your productivity

Open space offices are kind of an antipattern. You can find relevant articles and studies easily. Quiet: The Power of Introverts in a World That Can't Stop Talking by Susan Cain can be a good starting point:

Open-plan offices have been found to reduce productivity and impair memory. They're associated with high staff turnover. They make people sick, hostile, unmotivated, and insecure.

This post is not about bashing open-space. The world outside of the office is not much better at promoting focus and deep work. Significant others, neighbors, cars, trains, birds, animals, generally humans and non-humans tend to make noise. Optimizing your environment for focus is hard and sometimes outside of our circle of control. This post is about doing damage control where we can.

Spacing out

Headphones - the international sign of a focused developer. Some buy 900$ active noise-canceling headphones, some just blast 80's power metal full volume. I'm not that keen on expensive tech gadgets, and music tends to grab by attention, which is just substituting one distraction for another.

A few years ago though, the Interstellar movie (yes, Interstellar came out 4 years ago!) inspired me to take a look at nature noises. Remember how they used to listen to a thunderstorm with crickets on the spaceship? So that's how I found...

MyNoise.net

tl;dr: check out that site, use it to nullify distractions, better than any 'programming playlist'.

On mynoise you can find noise generators that let you relax, meditate, sleep or not go mad during a 23 year long solitary recluse in a spaceship. Moreover, some of them are marked as noise-canceling - put on your headphones and hear what happens when someone talks next to you. White noise is the secret sauce that drowns the world outside of your headphones.

White noise

If you look up white noise, you are most likely going to find thousands of videos and applications designed to help babies sleep. Instead of running a hair drier or vacuum for 6 hours you can just play it on your computer, pretty convenient I guess. But what is white noise?

In the simplest terms, white noise is a random signal with equal intensity at different frequencies. If you look at the power spectrum of synthetic white noise you can see something like this:

If we merge it with a human speech the result is not always perfect; if you listen carefully, you can still understand some of the words. However it's usually good enough to prevent distractions. Consider someone talking directly in front of you:

And the same with applied white noise:

If the voice was generated two meters away - it would be utterly indescribable.

The topic is quite complicated, but the key takeaway is that white noise is excellent in masking other sounds. Moreover, some of the natural sounds are very close to white noise in the audible spectrum - rain, waterfalls, hair dryers.

I've been using mynoise.net for years already. Listening to actual synthetic white noise isn't my favourite way to use it, but natural noises that are 'close enough' are great for my focus - they helped me work or think even in the most annoying conditions. My favorite?

Tropical rain, Irish Coast, and Cafe Restaurant

Chatbots - should I call LUIS?

2018-07-05T00:00:00Z

Radical changes in Microsoft BotFramework v4

Microsoft stack for bots lets you quickly build smart and functional bots. It's great, check it out, build your own ~friends~ virtual assistants. Architecture can be boiled down to:

Bot Service - a connector that abstracts any channel (Skype, slack, etc.)
REST-ful app that you develop with BotFramework SDK that is called by Bot Service with every message
Any additional services that provide "intelligence" to your bot (LUIS, QnA, Azure Search..)

LUIS, part of that offering - Language Understanding service - enables you to easily detect what the user wants. If you want to know that user asks to "BookFlight" (intent) and you expect people to say something like:

    Book flight to [place] on [date]
    
    I want to go to [place] on [date]
    
    On [date] I want to fly to [place]

Congratulations, you have just pretty much written down a LUIS model.

Now, if you are interested in BotFramework you may have noticed that in SDK v3 the default behavior is to send every user query to LUIS. All samples related to intelligence use middleware that hijacks the flow if an intent is found in user answer.

Should we call LUIS on every single message?

The short answer: no. And that's also direction SDK v4 has recently taken, but before that let's consider two situations:

Where do you want to pick up your order?

At our restaurant
At your office
At your home

home

What's your home address?

Home Street 1145

Asking LUIS by default wouldn't provide any value. And on the other side we would dent our LUIS calls quota (50 calls / sec / service, chargable) and possibly get sensitive user data in endpoint logs!

SDK v4 - changes

BotFramework SDK v3 is the current production-quality release. V4 is in preview and changes rapidly, but if you look at the docs and samples you can still see LUIS middlewares used everywhere.

Well. 27.06, pretty much a week ago, all middleware for LUIS and QnA was removed from master branch. The idea is that we as developers must own the decision when to call LUIS. And that's great!

LUIS and input

In contrast to previous samples, let's consider:

What's your street address?

Wait, why do you need my address?

In that case the user obviously (for me and my fellow humans such as you, my dear reader) wants to do something different, but at the level of input to an app - we don't really know that. Should we call LUIS just in case?

The pragmatic reality

One of the most important (and far too often not used at all or poorly implemented) patterns in user experience is the ~undo~ functionality. However, as hard as it's to implement sensibly in conversational interfaces, we can at least rely on double confirmations

We will send your package to "Wait, why do you need my address". Is that correct

no, go back!

Confirmation might be the right place to call LUIS, or even better - try checking the answer with rule-based, "less-intelligent" parser.

Building chatbots is a very different kind of challenge that any traditional UI. There are hundreds of weird edge cases like that and at the end we are still nowhere near pretending to create human-level experience with chatbots. But if we as creators don't overpromise - users turn a blind eye for some minor blunders in our "intelligent" applications. And get some real problems solved behind the scenes which is the only thing that counts in the end.

Still, some brave souls that use preview version of BotFramework SDK may be in for a surprise after next release.

Automated page rendering revisited - Puppeteer edition

2018-05-18T00:00:00Z

The king is dead, long live the king

PhantomJS - default headless browser used by many automatic testing solutions - is dead. The only information about that fact can be found in one of 1823 open issues in git repository. Don't blame yourself if you didn't notice. Hopefully you haven't recently started a new project using it or spent few days writing an extensive post detailing workarounds for some interesting edge-case uses.

[EDIT: http://phantomjs.org/ is already updated with this information]

However, it deserves a moment of silence. PhantomJS had a good run, helped solve real development issues and even large brands like Netflix, LinkedIn and Twitter mentioned using it. Who knows how many critical bugs were caught thanks to it?

In the meantime, somewhen last year, Chrome shipped version 59 supporting headless browsing. There is not much to discuss - it's hard to ask for stabler engine than chromium.

Controlling your puppets

Google ships a framework for controlling Chrome/Chromium under a well chosen name of puppeteer. I have to admit I'm having a blast exploring its API and that's why today we are going to redo some of the functionalities from my previous post.

Puppeteer has also been featured on Google I/O '18, you can watch the session here and see the samples here

Typing your types

This time we are not bound by PhantomJS engine so we can create a regular Node.js app. I'll be using TypeScript - if you haven't used it before, it's a less-bad JavaScript that gets transpiled to JavaScript but pretty much looks the same like JavaScript. Sarcasm aside, it really helps with whole classes of issues and has a very low starting barrier for anyone proficient in JS and object-oriented paradigm.

I won't be going into details of TypeScript here - samples below barely use any significant features and you should be able to read it just like JavaScript.

Simple rendering

import * as puppeteer from "puppeteer";

async function run(): Promise {
   const browser = await puppeteer.launch();
   const page = await browser.newPage();
   await page.goto('https://mostlybugless.com');

   await page.screenshot({ path: 'mb.png' });

   await browser.close();
};

run();

Easy!

Viewport and emulation

Our end-goal is to verify if a website looks good on most common devices. We can easily set viewport of a page with

page.setViewport(viewport)

Besides width and height, it supports additional settings which I'm going to ignore

But one thing to keep in mind is that puppeteer provides

page.emulate

which you can use as a shorthand to emulate any of the already defined devices. Fun!

Single line sanity check:

await page.setViewport({ width: 1366, height: 768 })

Scrolling

In the previous post I committed some serious crimes against code to create gifs of page scrolling. With full node support this time we could easily integrate one of the more reasonable gif-rendering packages and call it a day, but one issue bothered me - you can't stop a gif. That's nice that you can notice an issue in the middle of a page, but if it's gone in next 200 ms, that's not a great user experience.

Now, puppeteer doesn't (yet) have a direct api to scroll, which is a shame. Still, since we can run anything directly on the page itself..

const stepPoints = await page.evaluate('document.body.scrollHeight')
       .then(scrollHeight => {
           console.log(scrollHeight)
           const adjustedHeight = scrollHeight - step;
           const lastScrollHeigthToRender = adjustedHeight > 0 ? adjustedHeight : 1
           const stepCount = Math.ceil(lastScrollHeigthToRender / step);
           return _.range(stepCount).map(stepNo => stepNo * step);
       });


   for (let point of stepPoints) {
       const path = `mb-${point}.png`;

       await page.evaluate(`window.scrollTo(0, ${point})`);
       await page.screenshot({ path });
   }

With this adjustment, we got

Summary

Puppeteer is a powerful framework and it's only starting to get attention. I expect to see many interesting use cases in coming months. Also, if you haven't yet tried TypeScript - give it a chance. It will be worth it.

This time I'm not going to completely automate the rendering as in the previous post, but maybe I'll come back to this topic. In the meantime you can get a good start with above code. Enjoy your puppets!

PhantomJS for automatic responsive page rendering

2018-04-05T00:00:00Z

PhantomJS is a headless WebKit scriptable with a JavaScript API

While you might've already used PhantomJS as a part of a test automation stack, it's also a powerful standalone tool you can use for fun and profit. My experiment today:

How can I automatically check my website on multiple screen sizes?

There are some great tools out there that you can try, but if you are running low-budget (like a personal blog) or feel an insatiable urge to overengineer a simple problem, feel free to follow along. We are going to end up with a report that will look like this.

Download PhanthomJS , I'll be running phantomjs-2.1.1-windows.
Though there is sadly no support for ES6, the API is really reasonable and well documented. Let's start with a sanity check and just render our webpage. Create a file render.js, which will load the webpage module and save it to png file:

var page = require('webpage').create();
page.open('https://mostlybugless.com/', function() {
    page.render('mb.png');
    phantom.exit();
});

Link to screen capture API docs

You can run it with:

phantomjs.exe render.js

Resulting image will contain full-lenght page with minimum-width render. Not quite what we wanted, but ~it's something~. 3. We can configure our script to change the viewport with:

page.viewportSize = {
    width: 1024,
    height: 768
};

Aaand that doesn't look like 1024x768. Viewport is the size of actual headless browser but rendering in default saves full page length. Let's fix that by letting PhantomJS know which part of the page we want to clip when taking screenshot.

page.clipRect = { top: 0, left: 0, width: 1024, height: 768 };

In some cases we may want to render full lenght of page, but as you can see on this very blog, we wouldn't get actual user experience for fixed position elements. Altogether we are now at:

var page = require('webpage').create();
page.open('https://mostlybugless.com/', function() {
    page.viewportSize = {
        width: 1024,
        height: 768
    };
    page.clipRect = {
        top: 0,
        left: 0,
        width: 1024,
        height: 768
    };
    page.render('mb.png');
    phantom.exit();
});

Tremendous success! We have covered the basic screen capture guide from PhantomJS documentation. We could easily for-loop it here and call it a day. Build package that you downloaded should even contains an example script by Salih Sagdilek - responsive-screenshot.js, which pretty much does that.

Let's go a little bit deeper

Rendering page with scrolling

PhantomJS API allows us to:

Evaluate any function in the context of opened webpage

var actualHeight = page.evaluate(function() {
    return document.body.scrollHeight;
});

Scroll page to a desired position

page.scrollPosition = {
    top: 200,
    left: 0
};

By combining those two features with simple math we can automatically save screenshots of a virtual full-page scroll to bottom.

var page = require('webpage').create();
page.open('https://mostlybugless.com/', function() {
    var viewPort = {
        width: 1024,
        height: 768
    }
    page.viewportSize = viewPort
    page.clipRect = {
        top: 0,
        left: 0,
        width: viewPort.width,
        height: viewPort.height
    };
    var actualHeight = page.evaluate(function() {
        return document.body.scrollHeight
    });

    var step = 200;
    var stepCount = ((actualHeight - viewPort.height) / step) + 1;

    for (var i = 0; i < stepCount; i++) {
        page.scrollPosition = {
            top: Math.min(i * step, actualHeight - viewPort.height),
            left: 0
        };
        page.render('mb' + i + '.png');
    }

    phantom.exit();
});

Now that we have got a series of pictures, how about

Making a page interaction gif animation automatically

If it was ever created, it will be rewritten in javascript at some point. It's true for git clients, web servers and video-processing frameworks - for better or worse. But in the latter we are interested. There is a small gif-creating library from yahoo we are going to use:

vgifshot

var page = require('webpage').create();
var gifshot = require('./gifshot.js');
var fs = require('fs');

var webserver = require('webserver');
var server = webserver.create();
var service = server.listen(8083, function(request, response) {
    response.statusCode = 200;
    response.setHeader('Access-Control-Allow-Origin', '*');
    response.setHeader('Content-Type', 'image/png');
    response.setEncoding('binary');
    var path = request.url.substring(1);
    console.log('request ' + path);
    var file = fs.open(path, 'rb');
    var data = file.read();
    file.close();
    response.write(data);
    response.close();
});

const GifReadyMessage = '[Injected] Gif Ready';

console.log('server started ' + service)

page.open('http://localhost:54359/', function(status) {
    console.log(status);

    page.onConsoleMessage = function(msg, lineNum, sourceId) {
        console.log('CONSOLE: ' + msg + ' (from line #' + lineNum + ' in \"' + sourceId + '\")');
        handlePageConsoleMessage(msg, page);
    };
    page.onError = logError;
    var width = 1024;
    var height = 768;
    var gifDownScale = 2;
    var gifFrameDuration = 5; // 10 = 1 second
    var scrollingPixelStep = 200;
    var fileNamePrefix = 'mb'

    var files = renderPage(page, width, height, scrollingPixelStep, fileNamePrefix);

    console.log('rendered')

    var injectsuccess = page.injectJs('./gifshot.js');
    console.log('injecting gifshot success' + injectsuccess)

    var imagesToUse = files.map(function(f) {
        return 'http://localhost:8083/' + f
    });
    console.log(imagesToUse)

    page.evaluate(function(width, height, frameDuration, downscale, images, readyMessage) {
        gifshot.createGIF({
            'images': images,
            'frameDuration': frameDuration,
            'gifWidth': width / downscale,
            'gifHeight': height / downscale,
        }, function(obj) {
            if (!obj.error) {
                _injected_tempGifStorage = obj.image;
                console.log(readyMessage)
            } else {
                console.log(JSON.stringify(obj));
            }
        });
    }, width, height, gifFrameDuration, gifDownScale, imagesToUse, GifReadyMessage);

});

function handlePageConsoleMessage(msg, page) {
    if (msg === GifReadyMessage) {
        var gif = page.evaluate(function() {
            return _injected_tempGifStorage;
        });
        var path = 'output.gif';
        console.log('Saving image')
        var gifContent = gif.replace(/^data:image\/gif;base64,/, '');;
        fs.write(path, atob(gifContent), 'wb');
        phantom.exit();
    }
}

function renderPage(page, width, height, scrollingStep, prefix) {
    page.viewportSize = {
        width: width,
        height: height
    }
    page.clipRect = {
        top: 0,
        left: 0,
        width: width,
        height: height
    };
    var actualHeight = page.evaluate(function() {
        return document.body.scrollHeight
    });
    var step = scrollingStep;

    var stepCount = ((actualHeight - height) / step) + 1;
    var files = [];
    for (var i = 0; i < stepCount; i++) {

        page.scrollPosition = {
            top: Math.min(i * step, actualHeight - height),
            left: 0
        };

        var fileName = prefix + '_' + width + 'x' + height + '_' + i + '.png';
        page.render(fileName);
        files.push(fileName);
        console.log('rendered ' + fileName);
    }
    return files;
}

function logError(msg, trace) {
    var msgStack = ['ERROR: ' + msg];

    if (trace && trace.length) {
        msgStack.push('TRACE:');
        trace.forEach(function(t) {
            msgStack.push(' -> ' + t.file + ': ' + t.line + (t.function ? ' (in function \"' + t.function+'\")' : ''));
        });
    }

    console.error(msgStack.join('\n'));
}

Oops, that escalated quickly. Combining quirks of a headless browser, gif-rendering javascript library that abuses DOM to achieve its goals, local files and, worst of all, javascript, was less then pleasurable. Taking it piece by piece, we:

Set up a webserver that serves static files from the script directory
Render page just like before
Inject gifshot into the context of a webpage
Request gifshot to create animation of locally saved renders and notify us through console.log-ing
Read the result and save locally in a hook to opened page console message

So, this ~maybe~ is a ~little bit~ hacky solution. I created this abomination for various reasons that came up, but mostly this is because gifshot uses DOM to do its magic and in this situation - access to locally saved pictures. Asynchronisity of gif creation and sandboxing between PhantomJS context and opened webpage did not help.

But it works! We are now automatically generating a gif of a full-page scrolling.

Now, we are pretty much one for-loop from doing a cross-spectrum responsiveness checkup. But first,

Which screen resolutions to test?

According to statcounter 80% of market share is covered by 20 screen sizes. We will as such apply the Pareto principle.

20 is still a pretty large number, but I will reduce some of them since height usually doesn't matter that much for overflowing content and some of them are "close enough". If your styling will break it will be either at smallest screens, largest screens, or right next to your breakpoints. I'm assuming the typical bootstrapesque 768/992/1200, so a following set seems pretty reasonable:

var screenSizes = [{
        width: 1024,
        height: 768
    },
    {
        width: 1280,
        height: 720
    },
    {
        width: 1366,
        height: 768
    },
    {
        width: 1600,
        height: 900
    },
    {
        width: 1920,
        height: 1080
    },
    {
        width: 320,
        height: 534
    },
    {
        width: 375,
        height: 667
    },
    {
        width: 540,
        height: 960
    },
    {
        width: 720,
        height: 1280
    },
    {
        width: 768,
        height: 1024
    },
];

Final solution

There is nothing interesting in looping over an array, nor doing minor parametrization or using again previous hack for passing data through a sandboxed page. However final version contains automatic report creation and folder handling which make it much more usable.

There is always a lot more that I could improve, performance and code quality, file cleanup. But I'm calling it Pareto-done for now. You can find the script here and sample report here, enjoy!

Notes and issues

PhantomJS is supposed to support ES6 from version 2.5, we are stuck now with the bad parts of JS.
PhantomJS is dead, project is no longer maintained. It will still be used for long time as it lies on the foundation of many testing setups, but don't count on issues being fixed. Take a look at headless Chrome and Puppeteer.
PhantomJS has bugs. Quite a lot of them actually. So before you cry wolf, make sure you are not seeing an issue with media query parsing.
Transparent background will be transparent on renders, and thus on gif, leading to really weird results. I can't imagine right now a valid case where your page's body would have transparent background, so just set it explicitly if it's not and you encounter problems.

Mostly Bugless

World Data Visualization Prize entry

World Data Visualization Prize #

The Big Idea: Predicting the Unpredictable #

Future of Frontiers #

Snapshots #

Contextualize predictions with your (or anybody else's) age #

Browse and explore future #

Search for specific topics #

Learn more, find more connections #

Get smart answers to your questions using LLM-supported chat #

Use social media and fediverse #

$ Following links are used to showcase webmentions functionality #

Hello (Again) World

The softer side of IT - public speaking

The case for public speaking #

Public speaking is not normal #

Public speaking matters #

It's just a skill #

The softer side of IT - effective learning

Effective learning #

Spaced repetition #

Willpower #

Tiny habits #

Unexpected synergy #

Links summary #

Mental models in software development

Time Management Knowledge Pill

Table of contents

Core Tenets

There is no such thing as time management

Time management is a skill

Main topics

Energy

Sleep

Exercise

Food

Managing energy

Habits

Overcoming bad habits

Making new habits

Keystone habit

Habit chaining

Bad habit or an addiction?

Mindsets

Meaning

Flow

The Eisenhower Method

Being busy vs being productive

Feedback and planning

Current state and Quantification

Goals

Objectives and key results (OKR)

Systems

Goals

Systems

Irrationality

Procrastination

Base tactics

Pomodoros

Time blocking

Environment control

Regular planning

Buffers

To-do lists

Getting things done

Ulysses contracts

Checklists and automation

Main myths

Visualization / self-affirmation

Working more

There is a single trick, 10 step guide, or a blog post that will change your life

Minor concepts

Resistance

Essentialism

Pareto principle

F*ck it effect

Inbox 0

Barbell strategy

Via negativa

World Data Visualization Prize

The Big Idea: Predicting the Unpredictable

Future of Frontiers

Snapshots

Contextualize predictions with your (or anybody else's) age

Browse and explore future

Search for specific topics

Learn more, find more connections

Get smart answers to your questions using LLM-supported chat

Use social media and fediverse

$ Following links are used to showcase webmentions functionality

The case for public speaking

Public speaking is not normal

Public speaking matters

It's just a skill

Effective learning

Spaced repetition

Willpower

Tiny habits

Unexpected synergy

Links summary

Habits

Apple and Google to the rescue?

A low-tech alternative

Peak bad habit

Noising up your productivity

Spacing out

MyNoise.net

White noise

Radical changes in Microsoft BotFramework v4

Should we call LUIS on every single message?

SDK v4 - changes

LUIS and input

The pragmatic reality

The king is dead, long live the king

Controlling your puppets

Typing your types

Simple rendering

Viewport and emulation

Scrolling

Summary

Rendering page with scrolling

Making a page interaction gif animation automatically

Which screen resolutions to test?

Final solution

Notes and issues