Analysing the freelancing market (ODesk & Elance)

[WORK IN PROGRESS]

Hi all, I am Jan and I am passionate about human-computer interaction, product design and entrepreneurship. I am currently reading for my PhD in Computer Science at the University of Oxford, where I am researching (together with the MoD and Airbus) ways to improve the situational awareness of our nations’ cyber defenders by conceptualising the management of cyber threat intelligence and developing technologies that support their practices.

I used Kimono for my research, to efficiently put a database of cyber security companies together that I then used to reach out to selected organisations for a potential collaboration (it was very helpful). Right now, I am experimenting with ways to help me track and analyse the freelance market (Elance and ODesk) in regard to supply and demand. This has nothing to do with my research and is just out of curiosity. I am interested in the concept of productised services and explore the usefulness of transferring some of the concepts of keyword analysis (from SEO) to this domain.

But on a bigger picture, I want to explore the usefulness of web data extraction approaches that support day-to-day/operational and also strategic decision making in SMEs (small and medium-sized enterprises).
I would love to talk to anyone who has an opinion, experiences or ideas in this area.

 

So here is what I did:

 

1. Find the pages that output “all results”

For Elance, this is https://www.elance.com/r/jobs/sts-0
/sts-0 displays jobs with “any status” and doesn’t limit the results to jobs which “hiring (is) open”. If I would only scrape jobs that are open, I will miss a great deal of data, since we will scrape on an hourly basis and jobs are often allocated within minutes.

For ODesk, this is https://www.odesk.com/o/jobs/browse/st/-1/
The same holds for ODesk. /st/-1/ displays all jobs independent of their hiring status.

 

2. Configuring Kimono (the web scraping application)

Click on the fields that you want to collect and give them a variable name. Easy. See here for help how to use Kimono.

 

2. Scraping the base data set 

I want to scrape all displayed results once (these will be around 47.000 results for ODesk and 26.000 for Elance). This will give me the data set that I will add to with later scrapes and mirrors the current job openings on these platforms. Important here is to set the pagination correctly in Kimono (here for more information) and set “pagination limit” to “10000 pages max” under the “CRAWL SETUP” tab of both APIs. And I had the “AUTO-RUN FREQUENCY” on “manual crawl” – I only need the base set scraped once.

The first crawl may take a while to run through. I looked at the results of this scrape and made sure the outputs were as expected.

 

3. Setting up the auto-run feature to track new jobs

To determine the crawl frequency in regard to pagination limit, I went to the Elance and ODesk pages and checked several times a day through how many pages new job posting from within the last 60 would stretch. On Elance, job posting from the last 60 minutes will stretch through an average of 6 pages. On ODesk, this is 1 page.

I went back to the “CRAWL SETUP” and changed the “pagination limit” to “10 pages max” for Elance and “1 page max”.

 

 

Implications for design (Kimono)

  1. Ability to set a custom number for the pagination limit.
  2. Ability to add-to previous crawls when using the auto-run frequency.

 

[WORK IN PROGRESS]

 

A hacker’s guide to finding startup jobs

THIS IS AN UNFINISHED ARTICLE

Getting a job, no matter if internship or full-time, in Silicon Valley is a dream of many. And it is the many that can make it difficult to get a job. Afterall, the job market is a supply/demand market. The less demand a job offer has, the easier it can be to get the job.

There are a few dedicated startup job platforms:

http://venturefizz.com/jobs For boston and NY
https://www.themuse.com/jobs?company=The%20Muse
AsiaWired.com – Looking for startups in Asia?  This may be the solution for you.
CoNotes.com – Focusing on nothing but jobs at startups, CoNotes has been around since 2007.
Dice.com – Browse jobs by city or pull up the category that applies to your skill set.
Jobs.Mashable.com – Our very own marketplace features categories for listing jobs and looking for them also.
NeoHire.com – Lets you look up jobs by category, add them to your basket as you find ones that interest you and then apply to all of the ones you’ve saved.
nPost.com – Besides offering numerous job listings at startups, they have 225+ interviews with people from some of the companies explaining what they are about and what they are looking for in an employee.
VentureLoop.com – Provides internship listings for students at certain schools and has job listings you can search by country or occupation.

While it is more convenient to simply go to Angel.co or StartUpHire and search through the database of job offers, there are many startups that don’t put their job openings on these platforms.

 

I will outline in the following an approach to find job applications that are likely to have less applicants – that are less advertised on job platforms.

We will circumvent the drawbacks of applying to jobs that are publicly advertised (e.g. on angel.co) by checking each Silicon Valley startup’s own job listings on their websites. But instead of doing this manually, we will automate most of the steps with the scraping tool KimonoLabs.

 

STEPS  Automatization Time needed
Step 1) Click through startup databases and find a company url automated
Step 2) Load the website, scan the website to find the page with job offers automated
Step 3) Analyse relevancy of job postings  automated
Step 4) If there are no job offers, search their email address for cold applications. automated For step 1-4: on average 30 seconds. For 1.000 startups that is 5.5 hrs

 

 

I am in my second year of my undergraduate studies and haven’t narrowed down my passion/career direction. I like many things, from product management, to marketing, to business development. Alternatively, in case you have a clear role you are looking for,

Step 1: Acquiring a dataset of companies

First, we will need to get a list of startups and their websites. We will use the databases of CrunchBase and Angel.co for this purpose.

To get a large data set of URLs of startups, we scrape a variety of sources.

Startup list curators: https://angel.co/companies?locations[]=Silicon+Valley&company_types[]=Startup

and https://www.funderbeam.com/startups?page=99999

We are adding an accelerator: 500startups: http://www.500.co/startup

And a VC firm: http://www.accel.com/#companies/2wire-acquired-by-pace

And a job search platform: https://www.google.co.uk/search?hl=en&q=site:http://www.ventureloop.com/ventureloop/companyprofile.php&gws_rd=cr&ei=1lY6Vfu4CorSaLL8gMgC&start=0&num=100

 

Note: The selection of sources is crucial for the usefulness of this whole process. Select the search parameters on each data source wisely (e.g. limiting to startups located in Sunnyvale). Of course, this is not just limited to startups. 

To scrape all these sources and export the URLs of our startups, we will use KimonoLabs. (a word on Kimono)

We now export all URLs and import them in Excel to combine them and get rid of duplicates and false websites. (Strip away “http://”, “www.” and “https:/” first and then get rid off obvious false positives such as *oracle.com/* and all URLs that link to sub-pages */*/. This is important and lead me to remove another 472 duplicates).

 

This leaves me with a final XXX startups that I have uploaded here – feel free to add new startups to the list!

Step 2: Scrape websites for relevant job offers

Key to this step Google’s search engine and an appropriate search query.

We will use the following search query to search each website for marketing internships, but experiment yourself with a few example companies to find the right search query:

site:*company_url*+marketing intern OR internship

 

Kimono scraping results:

 

Limitations:

It will not show job offers if the startup is exclusively using an outside provider to list the job offers (e.g. silkroad.com). Since we are looking for early stage (seed and series A, maybe B) startups, most will post their job offers on their website. Hence, this “limitation” may actually work in our favour. We won’t see job offers of any established big players.

Alternative A: Search for email for cold applications

site:*company_url*+jobs@*company_url* OR careers@*company_url* OR founders@*company_url*

Alternative B:

 

 

Bottom-line

I took away two things from this weekend-project:

  1. Web scraping (i.e. the automation of intelligence gathering) is powerful for all sorts of decision making. Kimono was helpful.
  2. Google, and search filters on Google and the individual sources is powerful and essential to know how to use.

 

The bigger picture: using scraping to automate day-to-day decision making

 

I plan on writing a blog post about jobs that “allow remote”, “location independent”, “worldwide”.

Also, set up Google Alerts https://www.google.com/alerts

Open Education at Oxford Entrepreneurs

Today I had a meeting with two brilliant minds; John Burk Stringfellow, the new President of Oxford Entrepreneurs and Yuning Chai, an Associate Intern at McKinsey and also a committee member. Oxford Entrepreneurs is the largest student society at Oxford University and the largest student society promoting entrepreneurship in the world. As of September, I will be honoured to become part of the committee and begin working on some exciting projects. One will be OE Podcasts, an initiative that draws upon the principles of open education. The concept for the upcoming online series came up when John asked me for ideas for the upcoming year. I pitched him the following high-level concept:

“Oxford Entrepreneurs’ (OE) mission is to inspire and educate on the highest level, and provide an outstanding network, competition and support. In order to achieve our mission and continuously become better, we are dependent on the following:

The Miracle of Customer Feedback

The reason I choose to work as a Community Manager at Gameforge back in 2011 was the ability to study the behavior of customers in the context of product failure. And that’s what it was, serving 300 customer tickets a day as the initial point of contact for inbound requests, I was supposed to “creatively and proactively assist users”. But apart from helping customers with questions, difficulties, bugs and trying to knock out negativity with sheer kindness, I also received once in a few weeks user feedback on the product.

At first I was agitated and upset when faced with negative feedback like this one: “The login process is completely counter intuitive! I am not able to login with my username! What are these guys thinking!” I was upset because the user was venting and had simply not realized that you need to login using your email address, because usernames in our system were redundant.

But now, a year later, working at a startup and proactively looking for feedback from users and investors, I realized the following:

If I had lined up our visitors per month shoulder to shoulder, they would have reached over 100km. Now imagine driving past at 50km/h, trying to look at each and every face swishing by one at a time. After 30 minutes I would be absolutely numb trying to see that many faces. Imagine how big that group is! Out of all of those people, how many are going to reach out to us and give us feedback on our product? Looking at the sheer size of that group, how many would contact us to complain if something was wrong? Even 0.1% would be thousands of people giving us feedback. Which clearly does not happen. Of all those faces, all those customers, it is less than one in a million who take the time to contact us to anything personal about what we are doing.

On the first impression, identical to the other requests, feedback does distinguish in the underlying motivation of the user. Unlike a particular action or information a customer is trying to receive by contacting us (e.g. getting to run the software after an update or to revoke an account suspension), the customer has no other intention but to interact with us when giving feedback (e.g. how our new product update completely sucks or telling about how awesome feature XY is).

This is a miracle.
Of all the things the coustomer could have been doing in that moment, like quitting the application or searching for an alternative product from our competition, he chose to interact with us. This is a miracle. The fact that he took the time to say anything, whether positive or negative, makes him extremely special.
In my role at Fair Observer, I am currently trying to get a grasp of our stakeholders perceptions to improve our product. Getting feedback is highly valuable, especially at our current seeding stage. We should treat any feedback we receive as a miracle and get a warm feeling when thinking about all of our disgruntled web site complainers.

Why and How to Use a Work Diary

One day you will look up and say – Hey, where did the time go? What did I do with my precious time? Not every body can answer quickly with great accomplishment. The people who can, have at least one thing in common: Focus. But being clear about short-, medium- and long-term goals is not enough. Getting feedback by reviewing your progress helps adjusting and optimizing your way of attacking goals.

One technique I discovered being useful is keeping a work diary. Personalities like Ben Franklin, John Adams, Andy Warhol and many others recounted at the end of each day what worked, what didn’t and some other random things that happened during the day. Reflecting your day will benefit your focus and patience and can help you to personal growth. You start to recognize trends in your work style habits and the factors that will lead to success or failure. Success is never triggered from a single event and a work diary can help to specify more clearly the reasons underlying its positive outcome.

There are no special rules for keeping track of your daily business, but I collected some useful tips:

  • SWOT: focus on strengths, weaknesses, opportunities and threats of your day
  • KISS: Keep it short and simple
  • Be honest with you. Failure is the key to success
  • Keep a positive mindset
  • Never delete what was once written

Therein is “growth power”, if pursued accordingly. It’s the trip afterall that gets us there not the beginning or the end. We first identify the goal but then we lay out the path and take the steps learning along the way by interacting and communicating in a positive way. If you face failure, take time to focus on what you can learn from those setbacks. Try to reinterpret those into something positive like learnings, practical experience or challenges. Setbacks are inevitable in complex work, so we should try to learn from them.

Existing cloud storage tools like Dropbox, Evernote and Google Docs (my choice) can help you accessing and writing your daily report from practically everywhere. Don’t forget to set your privacy status to private.

Image courtesy of Flickrcentellanta