Now that my first screencast has been finalized and published online, I thought it would be a good time to share insights into the things that I learned while producing it. You’ll see that what initially seemed to be a pretty straightforward task, turned out to be a quite a bumpy ride indeed.

The first, and probably biggest mistake I made, was to assume that the production of a screencast resembles anything close to the prep and delivery of the regular training sessions that I normally conduct. Talking to a live audience is much more forgiving and natural than making an impersonal recording. For me, it felt like the screencast uncovered all the weaknesses and dark sides of my communication abilities. 

Despite your best efforts, it is very difficult to deliver a natural-sounding script during the screencast. I eventually realized that the best method to use was to focus more on what I wanted to demonstrate, in terms of individual steps and important points, than to try and meticulously plan out a script. 

I would advise to start recording raw version first. Focus on showing the information that is important and try to talk it through set-by-step. Don't worry about saying something stupid, making mistakes or mumbling: that's all very natural. Recording made me realise how difficult is to focus on doing something while speaking about it at the same time.

When you’re happy with the recording (which might take several re-edits), review what you said and make appropriate corrections. I think that the best way to approach this is to write it all down. The trick is to use the edited version as the basis for your voiceover; it will make your speech much more relaxed and natural sounding and you don't have to worry about timing, because as long as you're watching the video track at the same time it's easy to get in sync.

I found that splitting both the recording and the voiceover into 3 - 6 minute long segments provided valuable checkpoints. Although the length of the screencast really depends on the target audience, it is worth noting that, in my (limited) experience and based on the first viewing statistics, it seems that the majority of viewers tend to give up after watching the video for 20 minutes or so. If you really can't get finish within this time, it's probably best to split the screencast into several parts.

With the right tools editing can be easy. My software of choice is ScreenFlow for Mac or Camtasia for Windows. Both share similar features like multi-layer editing, trimming, audio and video adjustments, callouts, and also provide streamlined workflow.

While the software you use makes the difference when editing, it’s the microphone you use that can be the real showstopper. My original voiceover was recorded using an everyday Bluetooth headset. Looking back I admit it was foolish to think it might work. The audio quality was just awful, and spending time trying to clean it up/adjust it was not worth the effort. I really would recommend that you buy a semi-professional microphone like Rode Podcaster; it is certainly worth the investment (especially if you're thinking about podcasting/screencasting on a regular basis).

Permalink | Leave a comment  »

Sorry, this post and the screencast is currently available only in Czech language.

Celý příspěvek, společně s dalšími informaci najde na Blogu DevOps česky.

Pokud vás screencast zaujal, a chcete se dozvědět víc o automatizaci infrastruktury a deploymentu (nejen web) aplikací, přijde na školení, které pořádám 1. - 2. listopadu 2011 v Praze.

Chef - Základy automatizace infrastruktury

Pro získání slevy 10% použijte kód SCREENCAST.

Permalink | Leave a comment  »

In last 10 months I became a terrible procrastinator. By the end of the summer, it had reached critical level. Whilst day-to-day work remained more or less reasonable, the long-term productivity imploded. The reason? A dramatic change in my daily routine and significantly different working style. 

Beginning in February, I stopped working 9 to 5, and, as a result the so-called 'normal working day' disintegrated into patchy fragments. The only way to get back on track was to find ways how to better organize my work and time to regain productivity.

Here's what helped me so far:

• One inbox became a necessity. Regardless of saying that the tools don't matter I found OmniFocus extremely helpful. Everything goes there - emails, short notes, tasks, errands, ideas - you name it. It doesn't matter if you rigorously follow GTD practices or not. Having one inbox guarantees nothing will slip out of the radar. Individual tasks might fall behind, but except in few extreme situations it's not a showstopper.
• Time boxing improved my ability to do one thing at a time. With Pomodoro technique it's easy to split everything into 25-minute chunks, short enough periods to overcome the temptation to do something different, and yet just long enough to get something done. After while I found that I tend to plan almost everything in pomodoros.
• Limited distraction achieved by opening email only during the time dedicated for it. Same goes for Twitter. And anytime I have a sudden, eart-shattering idea, I simply add it to OmniFocus inbox, where I could get back to it later.

  Still, I found sometimes, that's not enough. The temptation to lose focus was always there. Until I learned to switch off Wi-Fi completely when not needed. Try it. You get used to it after while. When you overcome the urge to Google everything life becomes much simpler. You begin to remember things again, and you improve battery life (laptop's, not yours) as side effect. 

• But, ultimately, the best remedy against distraction proved to be pen and paper. I bought some Pukka A5 manuscript pads and a nice roller ball, and Eureka, it worked! I re-discovered that scratching notes and concepts down on paper is fast, efficient, and much easier to visualise. Also returning to previous notes feels natural. 

  Initially, I scanned almost every handwritten page straight to Evernote. Now, I trust the paper even more and scan notes only from time to time. Event this post was originally written by hand.

All in all, these little changes helped me to get back on track, despite my challenging daily routine. That's something I can work on later… :) 

Permalink | Leave a comment  »

    It is not the strongest of the species that survive nor the most intelligent, but the ones who are most responsive to change.

    something Charles Darwin didn't say, but everybody thinks he did

First things first. Let's accept this - we're not so clever. That's why we suck at managing the change. And don't be afraid to admit it. It's natural. Who pretends otherwise is just seeking ways how to cushion the awkward truth. You might cover it by fancy frameworks, hide behind endless bureaucracy or blame somebody else. Your choice.

Now, if you compare the Clarence Darrow's quote above (yes, he's the real author) with the traditional understanding of the change management you might realize there's a significant difference. Whereas one is talking about the ability to adapt to a change, overseen or not, we are continually witnessing a desire to improve the likelihood of success only.

In context of the software development, there's only so much testing you can do to insure the successful release. And don't get me wrong - I'm not trying to suggest to test less, quite opposite. But there's always only a limited coverage you can achieve, either due to a budget constraints or sheer ability to predict uncertainty in the complex systems. Then you deliver. And it hurts. How do you manage that?

The difference might be a way how you adapt. We're already establish there's always going to be some element of pain behind change. Wouldn't it therefore be better to change the way it hurts? Just ask yourself following three simple questions:

   1. Who suffers by the change? Is it the developer? QA? Operations? Customer support or customer itself? The farther away from whoever the initiated it the less probable is to learn from it. Each step you introduce between the pain source and target will result in some information noise.
   2. How easy is to write a new code (i.e. introduce the change)? Are you new starters (and definitely not your regular developers) afraid to change things because they are afraid to break them?
   3. How fast you see something went wrong? You need to connect the change itself with negative reaction, exactly as you would do with your dog :) There's no real learning happening in something that happend weeks or months back.

For me that's about change of internal culture. Continuous deployment calls for responsibility, and you won't get it by removing the experience of value creation from your core business (developers for that matter).

Or you can look for yet-another-framework, introduce more bureaucracy or blame somebody else...

* I would be more than happy to quote original author of the three questions as mentioned during Devops Days US 2010, but unfortunately don't know who was it. Please, let me know if you do.

Permalink | Leave a comment  »

Goods require shipping. Software should not. Despite the IT industry successful move to detach the release management and need to physically transport the product, the ‘shipping’ is still creeping into the software development nowadays. Well, it’s even worse - it got embedded into our thinking. We don’t release, we ship. Those two became synonyms. What’s wrong with that, you ask?

Shipment of the product by itself doesn’t add any value to the customer and represents direct cost for the software vendor. We can call it waste. As we are taught the best thing we can do to eliminate waste is to try to reduce the activities where such a waste is produced. So, let’s not ship everything. Yes, the obvious and safe bet is to bundle as many features as possible to a single release. Iteration cycle time is getting longer and therefore expensive. If you want to bet the company’s well-being on a single (even though repetitive) milestone, you better invest some serious effort to make it worth while. That means testing. The safety net of testing of software is focused on improving testing coverage. Unfortunately as your product grows, QA has to do exactly the same. Overhead is increasing. Just when we assume we have reduced one type of waste we actually introduced the new one - waiting. Now we got two types of waste. Waiting and shipping.

Wait a second! Didn’t we just reduce the impact of the latter? Yes, but only at the end of the release cycle. In the process described above you still have to ship the product, and every time you do so waiting pops in. Waiting before the development team reach the agreed milestone, waiting to test the features, waiting to deploy the product, waiting to educate the user.

If you feel the pain of the process we’ve just described you’re on a good way to start fixing it. Luckily for us there’s already a way out of this mess. Internet not just significantly reduced the shipment cost incurred by a software vendor, it transformed the industry altogether. New vendors became service providers. Subtle change you might say, but it comes with a great simplification. Where software house ships the product to a thousands (or millions) of customers, service provider does it only to one - its internal operations team. No more hard to predict environments, no more unnecessary delays incurred. The pain has moved to a new level, from internal users to external ones.

Close proximity to the customer has already been accepted as one of the main advantages behind Agile movement. Shorter iteration cycles enabled faster feedback, automated testing reduced waiting time between development and testing. Product creation phase started to move faster. Stories got shorter, iterations more aggressive. In such an environment you have to pull down any obstacles to make the flow as continuous as possible in order to eliminate waste.

Traditional paradigms of the software development are now holding one last wall to conquer - deployment itself. In a situation where you have only one customer (you) there’s no more space for a separation of concerns. If there’s no cost behind operations talking to development do you really need to keep them apart? Most likely not. If you already automated most of your product life-cycle, don’t you want to conquer the last mile and automate the deployment as well? Yes! Move from continuous integration to continuous deployment.

I accept it might sound scary at first, because you we are effectively going to remove the last safety net, but you are also going to remove the separation between ‘us and them’. Developers will need to start thinking about every single line of the code in the context of deployment, start thinking as Operations team. And not to stop there Operations will need to start to behave like developers. Both sides have a unique processes and experiences learned over the time that can help the other camp.

And no, you can’t just wake up one morning and switch to continuous deployment. You have to work towards it, reduce waste over time. The implementation of smooth product flow will naturally exposes bottlenecks, expose the quality problems and therefore naturally leads to a reduction of the waste.

Let’s create software, not a new ways how to ship it somewhere.

For non-believers and pessimist the concept of continuous deployment isn’t really something new and coming out of a blue. It’s just a naturally extension of something called continuous flow applied in manufacturing and if you are interested you can read more about it online - Lean manufacturing.

Permalink | Leave a comment  »

We talk about it all the time - agility. At GoodData it’s in our DNA. We understand being agile is not necessarily easy. You need the right people, attitude and tools. And exactly as we’re trying to deliver the GoodData platform to help our customers bring agility into a BI arena, we need the very same tooling internally to sustain the pace of a 14-day release cycle. We bring our customers new features every two weeks - unheard of in the stodgy world of BI and analytics.

Let’s get some perspective since 14 days might not sound so difficult for agile developers. How do you stay confident that all 2,713 data marts, 1,344 dashboards and 19,086 reports (our internal count as of June 2010) are valid from release to release? For a long time, we performed very expensive and time consuming validation directly with our customers. But how many reports you can actually test? Where does the attention to every single detail end and become a boring chore?

It turns out it’s at the second report you have to validate. Then you have to add security and privacy concerns. We motivate our customer to be as self-service as possible, so why we should destroy this experience by a need to manual verification? GoodData is about the Cloud. And the cloud need automation to scale.

A few weeks back, we quietly and successfully deployed our Regression Testing Toolkit. Its purpose is simple: take two different code branches and select data mart(s) you want to compare and bang - results. All anonymized, with no data being exposed, and still providing enough information for our engineers to fix the possible problems. For me and you, the results might look a bit sci-fi but engineers love them, which is critical if they need to resolve the problems as fast as possible.

Here’s what it looks like:

As the devops supporter, I’m glad we found a way to get our engineers more engaged in our production environment. We got also a new performance tracking tool as a by-product. For every single report, and as aggregate for whole data mart, we now have a runtime value that is easy to compare across the releases or different environment. That’s what I call getting the good business value from our engineering team!

No matter how passionate we’re about this tool, there’s more work needed. And hopefully soon you will be able to see validation of your data and model directly. No delays. Enable you to iterate through the changes in your data model faster.

Building confidence and trust for on-demand solution is a long run. Tools like our Regression Testing Toolkit helps us to go that extra mile and me, an ops guy, help to sleep better…

Permalink | Leave a comment  »

Priorities do change, but old habits die hard. The luxury of having more than 10 or 15 hours a week for my personal software projects is now history. To keep up means to adapt. In last 6 months I’ve embarked on a personal quest to protect the last development activities and to be able to stay up-to-date with the latest technologies. This is a practical review of what I’ve found out and what works for me (so far).

Plan ahead the software project you’re working on. With the right milestones it’s easy to track the progress and avoid unnecessary distractions. There are hundreds of ideas you can always implement later. But somebody has to use the software first. I started to make a list and every time a feature comes to my mind as something useful I do increase it’s planning priority. Review it once a week, during some dull activity (like daily commute to work), to asses if the features have real benefit for the productivity or project effectiveness - reprioritize accordingly. In most cases 99% of features end up on ‘one day’ list.

Pick the right tools, because it’s nothing more annoying than to wait 10 minutes for Maven to download all the packages, and 3 minutes for your container to start/refresh application. Actually, in last year or so, I’ve completely abandoned Java as a platform and after some surprising turnaround started using Ruby. While working on a automated processing backend for Le Bebe (czech only; website is run by somebody else) I’ve finally understood the real power of Rails. Now I’m starting to be Ruby addict.

Don’t do something you can buy somewhere else for a good price. Going back to the Le Bebe, I’ve only written the engine to automate the sales handling from a different sales channels. E-Commerce solution is nowadays so cheap, it’s better to pay $30 per month or so and get it as a service. Even when using Rails, I can’t really say I would be sane to value hour of my life for about $1.50 - given the service fee $30 per month / (4 weeks * 5 hours I have a week).

Document, Backup, Version everything. Yes, Planning for 5 hours a week software development activities can be challenging and in some cases I don’t get back to the project for as long as 2 - 3 weeks. With all what’s going on with my day job, I’m effectively starting to loose the track of the details after couple of days. Only now I’ve learned to document all my commits (Mercurial and BitBucket) properly, and document every single decision (Google Documents).

Permalink | Leave a comment  »

UPDATE 2009-10-12: I’m happy to let you know this post is not longer relevant. Amazon AWS team successfully deployed the fix and the scenario used to simulate Denial of Service attack using UDP flood isn’t applicable anymore. All that in less than 24 hours after publishing the link on Twitter. Good job!

Original post follows.

Unfortunate events surrounding the DDoS attack against BitBucket kicked-off heated discussions about the nature of this vulnerability. Where Amazon officially acknowledged this to be a single isolated incident, many others started asking questions why did it happen in first place?

• Was BitBucket’s security group configuration set to block UDP traffic?
• How come they haven’t got better visibility of the on-going attack?
• Is this really Amazon’s fault?

Both personal and professional interest led me to find out more. Having designed series of tests how to replicate this scenario, I’ve started first instance and set up the target environment.

instance : c1.medium (us-east-1d)
EBS volume : 200 GB attached to (/dev/sdf)
monitoring : vmstat, netstat, iptraf, Amazon CloudWatch
security group : allowed SSH only (port 22/TCP)

UDP flood set up to be generated from the second instance (c1.medium) using simple Perl script, managing to generate whopping traffic of 650mbit per second (according to iptraf) using 1KB packets to random ports on the target IP.

Test 1. Let it run has been successful in a way there was no visibility on target machine. Still surprised by the traffic level generated on the source box, I’ve pointed the UDP flood to another machine – with security group allowing UDP traffic (ports 0 – 65535) – to check if the network traffic is able to reach another box. And it was. Not only from the same availability zone, but even from the different ones (tested us-east-1c and us-east-1b).

Test 2. consisted of formatting the prepared EBS, 5 samples for both scenario with and without UDP flood.

No traffic (1m15s)
UDP Flood (2m54s)

During the test there were only moderate increase in IO waits (somewhere between 2 – 4%).

Test 3. Bonnie++ performance test of the EBS volume. Running with no incoming traffic, it took around 8 minutes to produce quite reasonable report. Having switched on the UDP flood I’ve repeated the same tests and my expectation was to see some results in similar time. Fifteen minutes later and bonnie still haven’t even finished third step (rewriting). Another 10 minutes without any significant progress pointed me to do some research what’s going on. The box wasn’t performing virtually any IO operations, and time spent waiting for IO topped 100% every second reading (1s delay). Bingo!

To verify if the problem is really caused by incoming UDP flood, I’ve stopped the traffic for a brief interval (around 7 seconds) and monitored using vmstat:

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
0  1 893480  11272   3112 1699240    0    0     0     0   10   11  0  0 66 34
0  1 893480  11272   3112 1699240    0    0     0     0    9    6  0  0  0 100
0  1 893480  11272   3112 1699240    0    0     0     0   10    9  0  0 67 33
0  1 893480  11272   3112 1699240    0    0     0     0   11    8  0  0  0 100
0  1 893480   8824   3100 1700052    0    0 23808 24864  962  697  0  1 68 31
0  1 893480  12284   3084 1697988    0    0 16384 16576  711  424  0  2  4 93
0  1 893480   9020   3084 1700088    0    0 20480 20720  817  563  0  1 68 31
0  1 893480  10432   3072 1700192    0    0 20864 20720  907  612  0  4  5 90
0  1 893480  10976   3040 1699724    0    0 15620 12432  588  423  0  1 68 31
0  1 893480  10872   3044 1698556    0    0 12676 16576  600  350  0  2  2 96
0  1 893480  10328   3024 1700676    0    0 19976 16576  761  535  0  1 68 31
0  1 893480  12408   3004 1698096    0    0  8708 12432  457  254  0  1  4 95
0  1 893480  12408   3004 1698096    0    0     0     0    9    7  0  0 67 33
0  1 893480  11636   3004 1699120    0    0  1024     0   38   38  0  0  0 100
0  1 893480  10548   3004 1700420    0    0  1280     0   47   45  0  0 66 33
0  1 893480  10188   3004 1700756    0    0  3584  4144  195  110  0  0  0 100
0  1 893480  10120   2992 1697968    0    0  6404  8288  256  205  0  0 67 33
0  1 893480  12468   2992 1696864    0    0  8064  8288  343  250  0  0  2 98
0  1 893480  11720   2972 1696984    0    0 12420 12432  495  333  0  0 67 32
0  1 893480  10136   2976 1700800    0    0  6916  4144  321  190  0  0  0 100
0  1 893480  11972   2956 1698820    0    0  4096  4144  161  117  0  0 67 33
0  1 893480  11364   2960 1699480    0    0  3844  4144  200  126  0  0  1 99
0  1 893480  11432   2960 1699480    0    0  2944  4144  160   91  0  0 66 34
0  1 893480  11156   2960 1699820    0    0   256     0   18   12  0  0  0 100
0  1 893480  10884   2960 1700020    0    0   256     0   17   17  0  0 66 34
0  1 893480  10856   2960 1700076    0    0     0     0    9    8  0  0  0 100
0  1 893480  10856   2960 1700076    0    0     0     0    9    9  0  0 67 33

As you can see on line 5 the IO traffic resumed, roughly correlating to the time incoming traffic stopped. Seven seconds later with the UDP traffic back on the box tried to keep up for another quarter of minute before giving it up.

Nothing! Based on my notes the first bonnie run occured at 10:40, switched on the UDP flood at 10:50, and started second bonnie run at 10:52. My patience ran out before 11:30 where there’s small peak caused by interactive iptraf session.

At this point there were no reasons to continue testing. All IO operations to/from EBS volume seemed to be blocked by UDP traffic generated by a single instance!

Conclusion

BitBucket guys had every reason to be angry. Blocking UDP in the security group configuration only hides the problem. Contraindicating the Jesper Nøhr statement, during this experiment there were no peaks visible using paid monitoring service – Amazon CloudWatch (see above). Which was probably the amount of information available to AWS 1st line of support.

This corresponds to the ‘black box’ described by Jesper. Looking back on the results it’s obvious that

• on-demand network capacity backfired in this case
• security group configuration is most likely applied on the host system
• host architecture seems to be sharing same network interface(s) for actual network traffic as well as network traffic to/from EBS instances. Even though instances got only a single network interface, I would expect this separation to be implemented on the host system. Segregation of the network traffic is one of the first lesson learned in high-exposed clustered environment.
• a week after the attack and there isn’t any fix in place. Hello, Amazon?!?!

To be fair, it’s been the first incident of such a magnitude. Let’s hope Amazon AWS team will come up with the architecture fix before somebody use the vulnerability in much wider and devastating attack. In mean time, the only workaround we can apply is to hide our instances as much as we can. Load-balancers and proxies in front of the worker instances should be enough, as long as you don’t share the same host machine.

Have a good weekend and good luck protecting your instance’s IPs!

PS: who had the same dark thought as I just had? What about S3?

[UPDATE 2009-10-11 7:00pm] c1.xlarge instances are able to generate UDP flood in the rate of 800 mbps. I guess, Amazon AWS is running 1Gbps network infrastructure.

Permalink | Leave a comment  »

Following the rocket like boom of the cloud computing by the end of 2007, countless questions have been asked about security aspect of such a solutions. For many businesses this concern may overshadow other benefits – like agility, cost effectivity or scalability. This post is my reflection on true considerations one should take into account when moving into the cloud; all in perspective of the small to medium size businesses.

Many articles and studies casted a dark shadows on the general idea of on-demand provisioning of infrastructure. And they are right in one perspective: if you are not able to provide adequate security measures to your local hosted data or solution, you won’t be any better in the cloud (well, almost). Added remote access will only exponentially increase number of the potential intruders. But where this shadows do not reveal complete truth is the fact the lack of security is very often given by inability or negligence in businesses itself to establish adequate security. Which is not only the problem of small business, as we learned last year (2008) by series of blunders going all the way up to the British government.

Going back to the security of the cloud offering, where increased number of the security threats is anticipated, providers are (hopefully) taking preventive measures in place which we, regular users, wouldn’t be able to afford locally, especially in situations where the expectation is to bear the upfront cost of such a protection – no matter if it’s a physical equipment or operations staff (it’s up to you to pick the one more expensive for your business). As there’s no vendor able to address all the possible aspects and requirements, many of them choose openness to allow partners to provider added services. Perfect example of such a cooperation is the community surrounding Amazon AWS. Service aggregators will and have already started filling in the missing picture.

Other reactions are continually disputing physical security of the cloud computing and how such an anonymous solution can replace traditional collocation, dedicated or managed hosting services. It may sound bold, but I feel confident to say not only it can replace them but it certainly will, unless their are proactive in their offering. Based on personal experiences with even the most reputable companies on the market today we have to accept the accidents do happen, always due to the good old human factor. Especially when operation support is focused on an individual and very often not related resources, rather than anonymous blobs managed as a whole. Luckily traditional hosting market is not sleeping and we can already see different cloud based services coming from companies like Rackspace (Mosso) on one side and UK2 (vps.net) on another. To polarize the opinions a bit more I can’t wait who will knock in a final nail in the coffin of the companies refusing to change by introducing hosting platform provisioning on top of the existing clouds.

Due to the varied nature of the different cloud computing services it would be outside the scope of this post to list all the different security concerns, recovery scenarios and long-term viability options. This make selection of the provider important task, but the point is the process itself hasn’t changed so much compared to what we already know. Cloud computing is changing IT as never before, but it’s not technical rules that are changing (they’re evolving), but the business model is where the revision is being done; the rest is just a reflection of it.

Permalink | Leave a comment  »

I love my Epson V750. It’s a remarkable scanner with price tag that won’t ruin your budget. For a little bit over £500 you can’t get anything better. With one exception, the original film holders are difficult too use and way to flimsy for scanner’s professional ambitions. Luckily, there is a solution for this problem - Dual MF Film Holder (120/200) from betterscanning.com.

After superb communication with Doug Fisher, I expected to wait eight days for new parts to arrive as recent huge demand depleted his inventory of components. Exactly as promised, nine days later I received shipment details and not very long afterwards Royal Mail delivered my new film holder and ANR Insert.

I won’t exaggerate if I say the film holder is everything I was hoping for. Its solid and very well made construction is easy to use. T-Lock insert system provides necessary support to keep film almost perfectly flat.

Many users of V700/V750 scanners reported unsatisfactory quality of scans. Unlike dedicated models there is no option to adjust and focus the scanner’s internal lenses. That may result in lack of sharpness in the final image. Original holder provides height adjusters to change default height between the film holder and document table for about 1mm.

Doug Fisher’s Dual MF Film Holder allows highly improved adjustment. I haven’t tried the maximum height you can achieve, as the optimum height for my scanners is between 1.6 and 1.8 mm. Although the process requires a bit of patience, it’s still pretty easy and once you get your focus right you’ll be rewarded by adequate sharpness.

Well, I can continue going through the rest of the features, but it will be just repeating what is already written on the original pages for this fantastic product. I haven’t got chance to try ANR Insert, but expecting it to be perfect for longer panoramic film.

If you have V700/V750 and scanning 120 films, don’t wait and get one. You won’t regret it. Only thing I can do is to express thanks once more again to Doug Fisher for very smooth transaction and perfect product

Permalink | Leave a comment  »