VisiData’s New Menu and the Power of Open-Source Collaboration

I first made a version of VisiData back in 2011 when I worked at F5 Networks. It proved so flexible and effective that I found myself missing it after I left. But because F5 owned the original, I decided to remake it completely from scratch and release it as open source. 

As an open-source tool, I knew it would get the most use if it was user-friendly and done the “right” way, which meant a lot of work on my part. And it was precisely others using it, appreciating it, and providing their feedback that made that effort worthwhile. My good friends over at GenUI are one of the sponsors of this ongoing open-source project, and recently with the encouragement of Shane Delamore, CTO at GenUI, a GUI-like menu system was added to VisiData in v2.6.

What is VisiData?

Simply put, VisiData is a Python-based interactive multitool for tabular data. It provides a spreadsheet-like interface for many formats and the utility of a terminal for easy exploration and arranging of the data. I like to think of it as duct tape, or maybe a Swiss Army knife, for data, because it is both sturdy and versatile. 

As a lightweight utility, VisiData can handle millions of rows with ease, allowing users to explore, clean, edit, and restructure the data in numerous ways, including rearranging and transforming columns, filtering, and grouping. It even has the ability to save, document, and replay workflows. 

The long list of possible uses includes the following:

• Easily explore datasets regardless of their formats
• Convert any input format into any output format
• Replace many command line interface (CLI) tools
• Scrape HTML data from webpages
• Parse and explore text-based log files
• Explore Python objects from the REPL
• Explore any pandas DataFrame
• Create ad hoc data pipelines
• Rename multiple files

In other words, it can do a LOT of different things! However, with so many different options, new users sometimes found it challenging to get started, and even experienced users didn’t have an easy way to discover new functionality. 

But even knowing that, I was resistant to creating a traditional hierarchical menu system. VisiData is designed to get straight to the point, avoiding the layers of UI that often get between you and your data. However, in my drive to create something new and efficient, I had missed an opportunity to make it accessible.

The New VisiData Menu System

Shane at GenUI provided the encouragement and inspiration for implementing a hard-coded menu system as part of the VisiData v2.6 release. Users have wanted a menu system for a while and I’ve made a few attempts, but this is the first time I decided to go with a traditional GUI approach rather than trying to invent something fresh.

At any given time, there are a thousand potential projects on the VisiData roadmap, so Shane drawing a line in the sand and telling me that a menu needed to be a priority is really what pushed this forward. Knowing that the menu would improve the accessibility of the tool gave me the courage to spend a month making it happen, and to focus on the product design aspect of it.

In the past, I implemented a sheets-based architecture for the menu system because I wanted to do everything in the same standard VIsiData paradigm. But it just didn’t work as well. I put the menu prototype in and then soon took it out. There has long been a command list that a user could search through, but it didn’t have the same comfort and ease of structured exploration and discovery of a classical menu. So, with Shane’s encouragement, I went with something that gets out of the way of the user and gives them a simple way to navigate.

VisiData now has a new menu at the top of the terminal screen. It’s a standard, clickable GUI menu that most users will find intuitive to use. The menu consists of VisiData commands and submenus. When you click on a command, it throws up a help box and you can see the help string for the command as well as the keyboard shortcut and the command long name. Of course, as with everything in VisiData, the menu can be explored completely through the keyboard; the mouse is optional.

The new menu system greatly improves navigation and command discoverability for both beginners and power users. I recently made a video showing the workflow for tracking which commands are in the menu vs not, with expand+melt+join, for those who are interested. Moreover, the design of the menu—with Edit, View, etc. in a bar at the top—also makes people feel more comfortable than a stark screen.  We’ve already gotten some very positive feedback from long-time users.

Always Learning

This latest addition to VisiData has given me some new insights. The first one is that sometimes it’s okay to do things the standard way. It turns out I don’t need to reinvent the menu system to add value to the tool. I had been avoiding creating something like this because I had elaborate ideas of how I wanted to do it. In the end, I realized I needed to look at it from the perspective of a new user and make something that is functional and intuitive. Although GUI menus aren’t the most elegant thing in the world, they are universally understood. That makes VisiData potentially useful to a much wider audience.

Building this new menu was like turning over the compost. Previously, some tools were buried underneath four other tools. Now they’re in the same bin and easier to find. Going through the exercise of organizing commands into a menu has now made VisiData more accessible and hence more useful. I even ended up using the new menu to rediscover tools I had myself forgotten about!

Collaboration with GenUI has already brought significant value to the VisiData project, and a few more similarly sized leaps could take it to the next level. As we celebrated 5 years of VisiData at the end of October 2021, we put together a roadmap for the next 5 years, and the future looks bright.

Try VisiData for yourself

Learn more about VisiData on our YouTube channel

Learn more about GenUI’s approach to collaboration

Discover S3P, Shane Delamore’s open-source tool for manipulating large AWS S3 buckets