Project 7 Development Log
One of the tasks in Project 7 is to log our experiences as we work on development. I am doing so in a journal format to capture my experiences.
[1/24/09] Thinking through questions in the demo, I've continued to play around with some different feeds in Excel. I found that if I can find a simple page of data, such as http://research.stlouisfed.org/fred2/series/PPIACO?cid=31, then I can pull this directly into Excel.
I've also built a page in our team wiki on the how I did my mashup. Not too technical and definitely not a cookbook. Just enough information to allow someone to know the steps I went through to build it. I need to include a version of the spreadsheet, but since it's a public page, I need to strip out some references.
[1/21/09] We went through our demos with the rest of the team this week and my demo seemed to go well. Now on to preparing for the presentation and summarizing my development experiences.
[1/7/09] As a team we are preparing to move on from building to preparing our presentation. Still an important part of development. As I look back on my feelings so far and my team's experiences, I've had this feeling that I haven't stretched enough because I haven't explored some of the technologies as much as others. For example, I haven't worked with an API such as one group is. But in terms of experiencing development in the tools, I'm thinking I may be harder on myself than necessary. I have learned a lot about what tools can and can't do. I actually spent quite a bit of time with Yahoo! Pipes and learned that as a tool for enterprise mashups, it doesn't quite fit the bill. I am glad that I sought out the assistance of Sarah on Pipes. Sarah is a developer by trade and has been the most motivated to make every technology we've touched work. She helped me understand more of the functionality. While it still wouldn't do what I wanted, I learned a lot. I think we all have learned that the actuality of mashups isn't quite up to the hype. Mashups are mostly used by "techie" people. Enterprises using mashup tools use a lot of IT support to make them seem so simple. The tools are coming around, but a small business should be more excited about the potential rather than what they can do today. All people should be thinking about this - "what else can I do or how can i do this better"?
So the question I'm reflecting on now is whether I should have done more development work or whether I've learned that relative to the charge - how a small business can use enterprise mashups - many tools are still more driven towards a consumer world. It still seems that much of what one would do with a mashup is focused on information consolidation, mapping or customer relationships. I have also noted that many ERP & CRM applications are adopting processes that we have been considering for mashups. So mashups as a concept will always change over time. From this angle, maybe I have stretched. I've thought more about what businesses need to run their business. I believe I understand the technology well enough to be able to tell someone what they could potentially do. There is a balance in this process. How much does one need to know to be able to advise or guide an organization. My goal was never to be a programmer. Working with things like API's is more on the how to build it side vs. the what can I do with it side. For my goals on this project (and hopefully the goals of the faculty as well), one needs to know enough about how it works to know what it can do. I think I've learned that on this project. I'll admit that some of this learning is coming from my observation of the efforts of my team. But I'm thinking my hands still got "dirty". Now on to the very important step of demonstrating the ability of what I have developed and showing one how it could be reproduced or modified.
[1/5/09] Looking at the rest of the deliverables for this project and there are some important items left to do: documentation! I've been thinking about recreating the steps for this project, but I'll need to work on it some more. Documentation is always difficult to think about while creating something. But anyone who has ever tried to recreate a project (theirs or someone else's) understands the importance of it. This is a big part of the training for an auditor. We are taught that the work papers from an audit should stand alone, since many times the person performing the procedures won't likely be there to discuss the procedures. (The analogy used to be that only the work papers would be on trial - but Enron and others brought that a little close to home!) Still, I need to make sure my documentation is up to a similar standard.
[1/2/09] The GoogleDocs feed import isn't working with my feed. I'm thinking that my table format isn't exactly right. I need to think about how far to go with this. The xml feed to Excel 2007 still works for what I do at the College. I wanted to work this up from different angles, though. Time to consult. I am sure I can use this format for other sites, such as a page or pages of items from suppliers. I struggle with developing processes like this in the abstract, though. I want some real data to work with and haven't come across what I wanted, yet. I'm posting a question to the team about how far we need to take these mashups.
Also maybe I will focus instead on other options for the xml feed, such as supply pricing.
[12/29/08] Working on the presentation has been fueling the thoughts of my team. In my case, thinking about the talk on Dapper made me think about what else I could do with this. This may be an even better angle. If Dapper can make a feed from a table of futures prices, then it could also make a table from any suppliers web catalog. It could allow comparison shopping from a list of preferred vendors. The Excel part isn't difficult with a few columns for suppliers and some VLOOKUPs to match item numbers. I think that if I I can make this work in GoogleDocs, then I will have a good mashup for the presentation.
[12/26/08] Since I laid out some next steps in the last log, I’ll address them separately:
- So far no go on the spreadsheet-less version. The Dapper feed wouldn’t load into IBM and Popfly doesn’t seem to have any options for data crunching. They seem to want to just present the data. Useful, but not for this application. As I think about it, the goal of the presentation is how to overlay enterprise data with public information. Why can’t Excel be a mashup tool? If we are talking with small business then most won’t know of this feature and definitely won’t know how to make a feed. I think I may have what I want. Now I need to think about the demo.
- I think for purposes of the small bus presentation it will be sufficient to just talk about what we can do. Show the web site, but no need to show the step by step. But for demo, I’ll have to rebuild this. On this front, I tried working with the Open Kapow tool to build a feed. I also thought I would try again with a different feed for IBM Mashup. Open Kapow doesn’t appear at first blush to work like the Dapper site. Dapper seems better suited to pulling data from a table. But I’ll look at it further.
- Haven’t done anything with this item yet.
[12/16/08] Rather than wait for the Lotus Mashup, I decided to see if I could pull my new feed into Excel. After some research I tried using the Get External Data function in Excel. I selected my new RSS feed from Dapper as the URL for the external data. This gave me a table that I could work with in Excel. And I did! I modified a sheet I use at work and was able to pull the RSS data in with a VLOOKUP function in Excel. Challenges:
-
Can we do this without a spreadsheet? Since Lotus Greenhouse was down, I started looking at Popfly for this, but it does not appear to be designed for this type of fee, or at least not to do anything with the data. Other options may be an online spreadsheet such as Google Docs or OpenOffice. But if this is necessary, don’t most have Excel?
-
I had to manipulate the data from the feed into something that Excel could work with. This required some deeper knowledge of Excel functions and some new columns added to the data. It’s not complicated, but most are unaware of the functions and most users to whom I’ve talk about Excel don’t think to work with data this way. Q: is this worthwhile for a demonstration? Or do we just talk about the feed and the ability to mash
-
I need to research whether the earlier versions of Excel could work. I believe that they will, but an extra step may be required using Google Docs, at least according to an article
[12/12/08] Developing the proposal is done. Concerns for me are more in the tool and learning for the project. I’m excited about the prospects of this project for my work as well as the concept of showing small business how they can use this without an IT staff or heavy use of a consultant. Proposal is up and now onto development. In the proposal I talked about the identification of the feed as the most difficult step of the mashup. Most of the first attempts found feeds that were fee based subscriptions. I then located a service called Dapper Dapp Factory that can create a feed from most web sites. I now have a feed, but the Lotus Greenhouse is again down. This is my preferred mashup app, so I’m waiting to see if it will be back up soon.
[12/2/08] Some success in working with development tools. Switched to Microsoft's Popfly product and the Lotus Mashup Server is back up. The Lotus mashups appear to be limited to Lotus Greenhouse users, but I could publish the Popfly mashups. This one maps customers from a sample database around Atlanta. (you'll have to zoom into Atlanta as there was one bad address mapped to the middle of the ocean!). Feel comfortable enough to think again about what to do with whatever development tool we select.
[11/29/08] P7 is well underway and after a much needed vacation (I took a weekend and went to Chicago), I'm to the point where I'm to get my "hands dirty" in development. This project excited me because it crosses my two specialties at E&Y: small businesses and IS. The problem is that I am more of a database guy and not a developer (see artifacts). So I've had problem working with the tool I'm evaluating. I'm working with two tools: Lotus Mashups and Yahoo Pipes. Big names that should work. The IBM/Lotus tutorials are very helpful and I was excited to work with this tool, which seems well documented for the inexperienced developer like myself. But the Lotus site is down and while I can upload my data to Pipes, I'm having a hard time matching it against anything on the web. Reflecting on what to do next: I'm not crazy about a couple of the tools that the team is evaluating either. I think my problems would be amplified in a tool that I don't have experience in. But we don't have to be solo, so I'm sure I can work through it. But I want to work this out. Time to look for more tutorials.
Added to this is that I've finally stretched this old PIII Gateway Solo laptop as far as I can. It's been a workhorse. It's not the old processor that is the problem as it is the 20GB hard drive (down to 3 GB of free space). Ordered a new Acer Aspire 4530, but the video was bad and it had to go back. So I'm soldiering on with the old one.