Keeping an Eye and a Thumb on Entrenched City Halls

As stated in a prior blog post, there is no plan to turn this blog into a data journalism site. There is still though a great deal of commonality between the efforts of new community paradigms and data journalism. Partnerships would seem to be a natural relationship. This is particularly true with new community paradigms efforts working from the outside, engaging with entrenched political institutions over a protracted period of time. This takes a more disruptive approach to establishing new community paradigms. Entrenched politicians often use anecdotal stories to support their decisions. Having a thorough analysis of the facts, especially those hidden from public view could help leverage change in a community. Even if a data-based story is not of larger publication interest it could still be of importance to the community in which it is occurring.

The first challenge is finding out what data there is available. This can depend upon how consistently open the government is to transparency and open data. Those which have an open approach can offer different methods of keeping current.

Some sites will offer “email alert” functionality. The Doing Journalism with Data, First Steps, Skills and Tool course offers UK government publications as an example, here is the American version. Others will offer RSS feeds (read more about RSS). The course features the Office of National Statistics in the UK and Data.gov.uk. Here in the United States, there is Data and Statistics | USA.gov.

This accessibility does not seem to be as prevalent though at the state and especially at the local levels of government and when it is available one needs to be sure that it is providing untampered and unbiased information.

Even when the institution does not offer either email alerts or RSS feeds, there are alternatives available such as ChangeDetection.com which can track changes and send update notifications with or without the willing cooperation of the institution.

There are also advanced Google search techniques that you can use to help determine what sort of information you want and where that information might be.

A Google search though can provide too much information and there can be a desire to focus the search. Searching with an exact phrase by using quotation marks e.g. “crime statistics” has fewer results compared to crime statistics with no quotes. Using a minus sign can exclude data, e.g., “crime statistics” versus “crime statistics” -national, the later will provide a more local and regional focus. You can also broaden your search with a wildcard asterisk "*". Using * for example within an exact phrase such as “crime statistics city of * ” will give you a list of websites of cities and their crime statistics. You have to experiment though and the number of results generated does not stay consistent.

You can also try searching within particular types of websites for specific information. Using site:ca.gov "community health” provides community health-related websites in California ending with ca.gov. Using filetype:xls will give you results formatted as Microsoft Excel spreadsheets, filetype:doc as Microsoft Word documents and filetype:pdf as PDF’s.

Searching for databases is a different matter as the contents of a database might not be visible to the search engine. In that case, you can use the word database "search by" instead of filetype. Databases though are the best means of organizing data for analysis if made efficiently accessible.

More advanced search operators such as Google’s advanced search, Google Guide advanced search operators page and the ability to combine different search operators are also available.

Sometimes, however, a possibly more confrontational approach, using legal avenues to obtain information by requesting it directly from government institutions through Freedom of Information or Right to Information laws is needed, when you cannot find it online. These are rules or regulations that provide citizens access to particular types of information, such as environmental laws.

To understand these legal rules and regulations check with sources for Freedom or Right to Information laws, or connect with people fighting for these laws.

You should speak with the organization in question before making the request, to check if they do hold the data but also anticipate possible exceptions and exemptions. Consider what judgments have been made on previous or similar requests.

The second challenge, beyond gaining access, is considering the means of analyzing the type of information you are seeking, whether it follows changes over time, requiring data over time or it compares items, meaning you need data dealing with the comparison or sometimes you need data that can add context to an issue.

All of this depends though on how the institution maintains its information online and how it provides access to the public. As suggested above, maintaining the data in machine readable form within databases is the best way of establishing a transparent open data environment.

If this is the case then you can request a related data dictionary. If investigating, for example, crime, gender, victim, and suspect information gathered by the police or investigating gifts or hospitality given to individuals by organizations and that information is kept in a database, then a data dictionary can be very useful.

These have data codes, designed specifically for combining datasets that use the same code. The course provides the HES online Data Dictionary Inpatients (PDF) from Great Britain. Here is access to data dictionaries for the United States Center for Disease Control and Prevention.

Obviously, it would be easier if all government institutions organized their data this way. This provides a good rationale for supporting open data laws. Often though the data is not so neatly organized and must be extracted.

There are means of collecting and extracting data usually left to those focusing on data journalism. Still understanding these methods so as to better work as or along with data journalists seems a good idea.

A primary means of extracting data is called scraping: using Google Drive spreadsheets or other online tools such as OutWit Hub, Import.io, Chrome extension Scraper or Scraperwiki. With a Google Drive spreadsheet which is available to just about everyone, it is matter of using the function formula =ImportHtml(“URL","query",index) with the URL of interest address in quotations, the type of data format which is either table or list you want, also in quotations and the sequence number of specific table or list that you want which is not in quotes.

As an example, the World Cup has been ongoing for the last few days so the course provides a FIFA webpage that publishes information on football agents in Cyprus. The formula for extracting the first table on the page is =ImportHtml(“http://www.fifa.com/aboutfifa/organisation/footballgovernance/playeragents/association=cyp.html”,”table”,1). Be warned though, you cannot just copy and paste from a Page or Word document or even blog post.

The New Community Paradigms wiki provides a variety of different of organizations, often through Facebook connections, outside the framework of institutional government that can provide knowledge and advice on a range of issues.

You need to map out what you’re interested in and which organizations, both governmental and non-governmental deal with that area. You might also seek other experts or pro-amateurs that collect and publish data by thinking about the people affected by the issue that interests you.

If looking for data in specialized areas of knowledge then you need to understand the jargon or professional language used in that arena. Sometimes you have to pick up the telephone and ask someone.

Data journalism pursues in-depth analytical stories that may be beyond the capacity or interest of an organization focused on particular community issues. There may be times though when statistics or data on a certain issue are needed to make a case for change in a community. There are a number of different types of sources of information available to build a case for change.

1) National statistical organizations

2) Local, regional and national governments and departments

3) International bodies

4) Regulators and auditing bodies

5) Charities and non-profit institutions

6) Corporations

7) Professional bodies and unions

8) Open data initiatives in your field

• Google’s Public Data page

• The Guardian’s datastore

• OpenSpending

At some point, these efforts take on the aspects of a long-term campaign by an established and organized group rather than one time issues. Maintaining a data library then becomes essential. The course suggests using social bookmarking services such as Delicious or Pinboard.in. This site uses the Diigo Group Page feature for new community paradigms to organize sites and related tags. You can also keep records of your sources in a master document (Google Docs, Microsoft Excel). Proper tagging allows you to find your data fast.

If starting to work as an organized group for change through new community paradigms, then the basic idea behind organizing your means of accessing, extracting, collecting and analyzing data in this fashion is the same as it is for data journalism.

"If you need to do something more than once, get a computer to do it for you," Charles Arthur (The Guardian).

Finally, even if the data is not available or not made accessible by a government institution or city hall, it still raises questions. Why is there a lack of data? What are the implications if certain data is not collected? What about data that exists but is not shared? These concerns still fall under the endeavor to create new community paradigms.

Data Journalism - Another Tool for Creating New Community Paradigms

Besides taking the ongoing STW/STIA Systems Thinking Certification course, there was also an opportunity to participate in a Doing Journalism with Data, First Steps, Skills and Tool course put on by the Data Driven Journalism site as part of the European Journalism Centre’s Data Driven Journalism initiative, a hub for news and resources from the community of journalists, editors, designers and developers who use data in the service of journalism.

The organizer is based in Europe but this blog has had no issue with adopting resources from Europe, Great Britain, Canada or anywhere else if it can help engage and empower communities. There is also a USA version of the course being provided through the Knight Foundation’s Knight Center for Journalism called INVESTIGATIVE JOURNALISM IN THE DIGITAL AGE, but it is much further along and was only just discovered so it will be covered in subsequent posts. The Knight Foundation was featured in a post from 2011 that is one of the building blocks for this effort, Finding the soul of your community and the reason to create your own community paradigms, more later on that as well. 

While there aren’t any plans on becoming a data journalist or having this blog become a data journalism site, there is still an important role that this type of information gathering and dissemination could play in creating new community paradigms. 

The first basic question addressed in the class is ‘What is data journalism?’ It is learning to use the techniques of data journalism (which changes all the time) to find the best possible way to tell a story using numbers.

Data, as  the course points out, has always been a part of how news organizations work. Historically, this also includes those using information to bring about change. Doctor John Snow's mapping of a cholera outbreaks in nineteenth century London changed how we saw how a disease progresses and serves as a model for data journalism today. Another surprising historical example, at least for me, is Florence Nightingale’s key report, ‘Mortality of the British Army’, published in 1858, which presented statistics on the Crimean War regarding war dead. The report documented changes in procedures at military hospitals initiated by Nightingale during the Crimean War and was illustrated using statistical diagrams.

Journalism’s job still remains reporting facts in a manner that people can understand more about issues that matter to them.  The added goal of data journalism is to bring numerical data to life making it possible not only to be understood, but I would add also to make it actionable. 

The basis for this ability is easy access to a variety of often free tools and other resources and their ease of use. This is one of the primary purposes of the New Community Paradigm Wiki. 

Today data journalism can reveal the numbers behind the news, on a national level as through the Associated Press,  US Election results 2012 interactive visualization. At the local news level, even with relatively little resources, it can tell stories that work for local communities, as  with the  Washington DC income gap (DC Action for Children). It can also be a watchdog on local community politicians, as the Texas Tribune did with it’s Ethics Explorer, A guide to the Financial Interests of Elected Officials.

This depends upon greater availability to open data though that is still an ongoing process. Hopefully, this will mean the development of a greater network of trust through increased transparency between those creating and generating data and those who are depending upon it. 

The skills required for data journalism are a collaboration between coding, designing, and journalism applied to a variety of different media products ranging from visualization to long form articles. It is the process of turning numbers into a story, regardless of whether the story is composed of words or of graphics.

There are different approaches to creating data journalism stories. The Lone Ranger approach is when you are able to do everything by yourself. This is possible because of tools now available such as those found in the Data Driven Journalism ecosystem. These include tools like OpenRefine, Datawrapper, Tableau, Google Fusion Tables, CartoDB. You could also have two person teams, like the Guardian's DDJ team in the USA that created the award winning Guide to gay rights in the US story or a small scale team capable of producing innovative projects quickly like the 'Flooding and Flood Zones' map Hurricane Sandy by WNYC.

It is large teams, like the New York Times that have the resources and the capacity to implement a deliberate strategy to create a new kind of online journalism. They can help with finding ways to tell the story better. An example is their 2012 Olympic Experience.

The size of the team is not everything though. Data journalism, according to the European Journalism Centre Data Journalism course, is about making friends which means it is about community.  The course is designed to  start out with smaller things, what that story will be, how to turn the numbers into stories and give you the basic skills to practice on your own or as part of a team with the potential of becoming part of a movement. The essential thing to remember is that anyone can do it. As the course states, you don't need to compete with the New York Times or The Guardian.  Starting small can lead though to bigger things.  Argentina's La Nacion, considered by many the best data journalism site in South America, started without a programmer using free software. What is important is the information. There are times when quick, messily created pieces can be hits having tremendous impact.  Even visualization is not always required for a compelling story. What is vital is getting the correct facts.

Data, the basic stuff, the building blocks of data journalism, is usually organized for use in commonly used formatted spreadsheets such as Microsoft Excel or Numbers. The key is the selection of the data. The fewer numbers you use to tell the story, the better.  This makes research the most important role but also the most tedious and time consuming, having to dig around in data on the basis of a journalistic hunch that may not pan out. 

If you adopt the Lone Ranger approach then you are going to have to do your own coding. If you have a team, particularly a small team, then your coders can also assist with research on the front end and visualization on the back end. A good team will effectively coordinate who can write and who can code. One notable example sited by the course is Reuter’s Connected China. Designers are those who can make visualizations happen. The example provide by the course is the Guardian’s 99% vs 1%.

At the end of the day though it is the words of the story that give context to the numbers, without context numbers are just numbers.  The Guardian’s Yearly guide to public spending by each government department  helps to explain the data.

“Public spending in 2011-12 was £694.89bn - compared to £689.63bn in 2010-11. That may look like an increase but once inflation is taken into account, it is a real-terms cut of 1.58%, or £10.8bn.”

I will finish up by letting Simon Rogers, one of the instructors for the course, sum it all up in this TEDxPantheonSorbonne video. 

“Today, data is increasingly accessible and simple to use, allowing journalists to develop a new way of sharing news. This revolution in the use of data is also accompanied by the increasing importance of data-journalists newsmakers, amateurs, representatives from the crowd ... No specific skills, unfailing motivation: the data-journalists are punks sharing of information.”