FLOSSmole

June data sets released

2009-06-15T14:20:00.003-04:00

Hi moles, the June 2009 data sets are released.

172-sf
173-fm
174-rf
175-ow
176-fsf

Datamarts (sql files) and flat (delimited) files are located on our Google Code downloads area.

oss2009 requests, etc

2009-06-08T10:19:00.003-04:00

Just back from OSS 2009 in Skövde, Sweden. (Finally figured out how to make the ö character on a mac: hit option-u, then o). Here are the requests I heard from sitting in talks, either for new forges, for features that FLOSSmole could provide, or just things that people were using/needing that might intersect with our mission here:

debian popularity contest
UDD
KDE's "10 years of data in an xml logfile"
sugarForge
ascencia?
eclipse
"git" everything
developer skills from sourceforge
rdfohloh
launchpad
gforge
fusionforge
dataportability.org
a wiki for common analyses, charts, graphs, SQL commands
a taxonomy of forges

May data, and April and May datamarts released

2009-05-03T13:50:00.002-04:00

Go grab the May data, April & May datamarts from our Google code web site.

I'm backing up to Teragrid now, so Teragrid users, you should have a nice new set of data to play with RSN (real soon now).

April 2009 data released

2009-04-23T14:47:00.002-04:00

Hi moles, April data has been released over to the google code site (all forges). I'll get that up on the Teragrid site ASAP. We have a few small problems with the scripts over there, nothing that is major but it will slow me down in getting the data uploaded over there.

In other news, we're getting a new backup system for the Elon servers in our continuing quest to keep the data safe. And I am pleased to announce that over the summer I will have another research student. She is going to be working specifically on user interface tools. Her name is Jamie and she's great. Look forward to a re-vamped web site, database schemas, and all that good "end user" type of stuff that I've been slack on providing up until now.

FOSSology grants

2009-03-08T12:11:00.002-04:00

Hello moles,

Here is an RFP I received this past week. I've copied it here.

Request for PROPOSALS
----------------------

Hewlett Packard, major sponsor of the FOSSology project
(http://fossology.org), is accepting short proposals from academic
institutions involved in research in areas of interest to the
FOSSology project. FOSSology is an open source project dedicated to
analysis and mining of open source data and using the results for the
betterment of the free and open source software (FOSS) and FOSS
management.

Interesting areas include but are not limited to:

- vulnerability analysis
- vulnerability tracking
- dependency analysis
- code reuse detection

Currently, the FOSSology project concentrates on software license
detection, but would like to expand into other areas. Feel free to
suggest your own area or propose something around the above topics

Grant amounts are in the range of $5,000 - $20,000 USD. The number of
grants awarded is dependent on the number and quality of proposals
received.

Proposals should short (1-2 pages).

Grant recipients will be expected to communicate with other FOSSology
developers through the public mailing list and/or IRC channel. If
grants are awarded on a single topic to multiple teams, those teams
will run their own program but be expected share information across
teams.

Submission date is open now.

Deadline for proposals: March 24, 2009

Grant(s) award notification (via email only) on or before April 30, 2009

To submit a proposal, send an email to bob.gobeille@hp.com with the
subject line "FOSSology Grant Proposal".

For additional information, please see http://fossology.org, or
contact me directly.

Thank you,
Bob Gobeille
Hewlett Packard
Open Source Program Office
bob.gobeille AT hp.com
bobg AT fossology.org
----------------------------------------------------------------------

Feb and March data released

2009-03-02T20:03:00.001-05:00

Sorry for the delay on February, but as a bonus I'll throw in the March data too!

Go to our Google Code downloads page and have fun.

Teragrid updated with Dec data

2009-01-22T10:26:00.002-05:00

I forgot to mention this here, but I finished updating the latest complete data sources to the Teragrid, so those of you who have requested database access should be able to play with that data there. The new data sources are:

150-sf
151-fm
152-rf
153-ow
154-fsf

Have fun! Let me know if there are problems or things that are just not quite right.

Dec data for SF released

2009-01-06T21:58:00.003-05:00

Sourceforge data is available for December. Get it now at the Flossmole Google Code Project Page.

Update (Jan/12/2009): Also datamarts from December are posted. Same place.

NOTE: The stats data looks somewhat strange to me. I suspect that the stats server was having some issues around the first week of December, so be careful of that data. I parsed the data as I found it, but I think it was not in great shape on the server when we collected it.

minor forges ready for December, inc. FSF

2008-12-05T09:52:00.003-05:00

The December "minor" forges are ready for download from the FLOSSmole Google Code Project Page: FM, RF, OW. Enjoy!

Update: I've also released Free Software Foundation (FSF) for October and December. This parser has been a long time coming. But better late than never.

Problem with Public Areas file

2008-11-13T15:01:00.003-05:00

We recently had an error in the Sourceforge Public Areas data dumps for October.

New files available for you to download. They are marked "2".

October developers

2008-11-06T09:06:00.002-05:00

Better late than never on the October Developers and Project Developers for SourceForge:

Download these developer files here

Sorry about the .gz format instead of the .bz - for some reason I have problems using the Google Code auto-uploader with .bz files. It's a very strange problem, random and intermittent, hard to pin down why some .bz files will go and some will not. Anyway, I gzipped them and we are good to go.

In case you never downloaded our developer files before, here's what is included there:

-- this shows the list of all developers
SELECT dev_loginname, realname, date_collected, datasource_id
FROM developers
WHERE datasource_id=143
ORDER BY dev_loginname

-- this links the list of developers to what project they work on, and also tells what role they have on that project
SELECT dev_loginname, proj_unixname, is_admin, position, date_collected
FROM developer_projects
WHERE datasource_id=143
ORDER BY dev_loginname

October data

2008-10-30T22:27:00.003-04:00

The October data are available at the following locations:

Google Code Downloads Page

Notes:

SF Stats are unavailable this month because the server was not reliably reporting stats when we did our collection.
There were a few errors in the file uploads. These files can't be taken off of Google Code, so I've just marked the files "DO NOT DOWNLOAD". You can usually tell these files by their very tiny filesize.
SF Developers and SF Developer_projects did not make the datamart build. I'll release these separately -- UPDATE: these are on our download site now

Bad August 2008 file

2008-10-06T12:53:00.002-04:00

The August 2008 SF file released to Google Code was corrupted, so anyone who downloaded this file only got 90k projects. Please download the new file here. It is called "Datamarts: SF Other, Aug 2008, New, v.1"

Direct DB access for FLOSSmole collection available

2008-09-09T11:23:00.001-04:00

Hello moles,

I'm excited to give you all a heads up that the entire flossmole database is now available directly via a MySQL server.

We have transferred the database to the NSF TeraGrid Data Central hosting site [1] (based at the San Diego Supercomputing centre). It's a bigger machine and professionally administered, which was much better than we could offer ourselves. See below for access procedure.

The process of transferring the database also enabled us to prepare comprehensive datamarts for each datasource in the database. These are mysqldump files which can be used for local access to parts of the database; there are two for each datasource, one containing the raw html pages and one, substantially smaller, containing just the parsed data points. These will be available shortly and will be an option for those who want to install a local copy of the DB; although we'd be very interested in reasons people find to do that, we'd like to have people sharing useful transformations of the data and the Data Central database should be pretty quick.

So now we have three great options for accessing the FLOSSmole data:

1. The traditional monthly flat files
2. Direct MySQL access to the full database @ DC.
3. Comprehensive datamarts for local access

Database access further info
------------------------------

In order to demonstrate usage to NSF and to monitor run-away queries (hey, I write them myself. Often :) interested users need to contact the FLOSSmole project to request a personal username and password, which should not be shared. Other than that simple request, we're not introducing any new AUPs or conditions.

Initially requesters should join and email their request to the ossmole-discuss list, with a preferred username. We can review using the list that way if the traffic spikes. Turnaround should be no longer than a business day or two (we email the db admin at Data Central with the request).

OTOH when we, and hopefully you, publish workflows using the database, we would like them to work 'out of the box', without a potential user needing to request a user/pass. To enable this, in addition to the full database we are in the process of creating a small database, with very limited data in each table (~20 rows in each table, just demo data). This is to allow querying through a single, public, shared login which we urge people to use when publishing their workflows; once potential users wish to go beyond the sample data they should request their own user/pass and plug it into the workflow. We're still figuring out the best way to do this (finding 20 projects with total data coverage is actually quite hard :)

Hopefully this improves the accessibility of the datasets, and will likely result in finding more bugs; both from the migration [2] and within the dataset. We're asking people to file bugs and request for documentation in the Sourceforge Trackers; although discussing them on this list is always welcome as well.

So, have at it.

--J

[1]: http://datacentral.sdsc.edu/ . The Machine (thor) is on Internet2 and has >80G of RAM.

[2]: If anyone wants to chat about ways to confirm data integrity while migrating 300GB+ databases with some very large tables, ping me :) I think we got it sorted, with the much appreciated help of our Master's student, Vinay Venugopal.

August data released

2008-08-12T15:05:00.003-04:00

Hello moles!

Lots of exciting news this month:

1. All releases are up on Google Code instead of SF this month. Go to the FLOSSmole downloads page on Google Code to pick up this month's files.

2. Sourceforge stats server is down, so no stats this month.

3. Debian is back! The data is minimal, but check out what we do have and enjoy that.

4. I've updated the Google Code FLOSSMole wiki also, so if you're looking for basic how-to or documentation, check there first.

5. coming soon: Free Software Foundation (re-released!), Eclipse (ask and ye shall receive)

ENJOY!!

Update: the SF file was corrupted, so anyone who downloaded this file only got 90k projects. Please download the new file here. It is called "Datamarts: SF Other, Aug 2008, New, v.1"

OSS Watch in Oxford

2008-07-21T05:55:00.002-04:00

I am at Oxford in the UK (staying at Hertford College Graduate Center) for a few days for the OSS Watch workshop on Profiling Open Source Communities.

OSS Watch is the National Advisory Service on open source for UK Further Education (FE) and Higher Education (HE). As such, it is part of our remit to help FE and HE institutions and projects who want to engage with open source development, and a key factor for that is the development of open source communities.

I'm giving The Standard FLOSSmole Talk today at 11:45GMT.

July data released

2008-07-08T23:25:00.002-04:00

Well moles, we were hoping to move to Google Code over the summer to host our file releases, but unfortunately, they have some size (and other?) limitations which are messing us up. They have no problem storing certain files, but others produce errors when I try to upload them. They are rejecting more and more of our files. I've put in a ticket to see if they can bump up the quota over there, but in the meantime, I'm back to putting files up on Sourceforge Downloads page.

Here's the download files area over on SF (available files for July: Freshmeat, Rubyforge, and Objectweb)

(Remember, since it's an "odd" month, for Sourceforge data, select the "June" release.)

I was able to get some of the files up on Google Code, so go over there and check the files out there if you like.

June SF data posted

2008-06-10T14:46:00.003-04:00

June Sourceforge data has been posted.

This time, I've released the files onto our Google Code project page. Let's see how we like using Google Code for this (psst, I can tell you as the person who does the file releases: it's a lot easier to use Google Code than Sourceforge for this particular part of the job!)

Enjoy!

UPDATE: Because of file size limits on Google Code, I've had to re-release our code onto Sourceforge.

Here is the link to the Flossmole SF Downloads Page, where you can get ALL the June files, including the huge datamart files.

May 2008 data released

2008-05-06T19:15:00.001-04:00

Moles, the May 2008 data has been released. Find it all at our SF project page. Forges included this month are: FM, RF, OW.

April 2008 data released

2008-04-16T19:34:00.003-04:00

Hi moles! April 2008 data has been released on our SF project page. Enjoy it! (Debian data included!)

UPDATE 08-JUL-2008: new files released to get past problem with data quality on these April files.

February data released

2008-02-09T12:57:00.000-05:00

Go to our file release page on Sourceforge and pick up all the latest files.

Included this month: SF, FM, RF, OW flat files and data marts (sql statements).

(Debian is on the way! I'll update here as soon as it's ready.)

Enjoy!

Jan files released and Dec datamarts posted

2008-01-11T10:11:00.000-05:00

Hi Moles!

The January data files have been released, and I went back and released the December data marts too (sql statements to make your own database).

Have fun with it! Grab the data from our Sourceforge project page.

December data released

2007-12-25T08:57:00.001-05:00

Hello moles! After considerable delay, here is the December data. Enjoy!

Download the data from our SF release page!

November data released

2007-11-04T16:42:00.000-05:00

Files for November 2007 have been released here:

Sourceforge File Release Page

October data released

2007-10-29T10:34:00.001-04:00

Sorry for the lack of postings! The data has been released, I guess I just forgot to post the updates.

Here are the October data links.