digiKam comes with a nifty batch utility that allows you to convert RAW files to the DNG format. The question is, of course, why you would want to do that. After all, digiKam can handle RAW files without any problem, so what’s the point of adding one more step to your photographic workflow?
As you might know, RAW is not a file format, but rather an umbrella term that describes multiple file formats controlled by multiple hardware manufacturers. The RAW formats themselves are specific to digital camera manufacturers. For example, Canon cameras store RAW files in the CR2 format, while Nikon cameras use the NEF format. Besides being proprietary, RAW formats are often poorly documented and encumbered by patents.
The Digital Negative (DNG) format introduced by Adobe Systems, Inc in 2004 is designed to overcome these shortcomings by providing an open, well-documented universal format for storing RAW files. If you would like to know more about the DNG format, the DNG articles and links Web site provides a wealth of valuable information on the topic.
Being an open and well-documented format, DNG is suited particularly well for long-term archiving of digital photos. Of course, the CR2 and NEF and other RAW formats are widely adopted and supported, so they are not going away anytime soon. But there is no reason why you shouldn’t save your RAW files in the DNG format to be on the safe side. After all, storage is cheap nowadays, and the DNGConverter utility makes it supremely easy to convert RAW files to the DNG format.
Using DNGConverter couldn’t be easier. Launch the utility, add the RAW files, specify a few options, and hit the Convert button. For complete peace of mind, you might want to enable the Embed Original File option which embeds the source RAW file into the resulting DNG file.
Tech writer covering Linux and open source software
Scribbles and Snaps 
That’s a nice tool. But I have two questions:
1. Is it only an GUI tool or do you have an command line tool too?
2. Is there somewhere a list with open source programmes supporting DNG? (Gimp, Thumbnailer, ImageViewer …)
http://www.bythom.com/dng.htm
Quoting Thom Hogan:
“there’s this thing some DNG proponents keep trying to scare people with: “some day your camera maker might no longer support your raw format.” So what? We already have open source code that interprets NEF (see libraw.org, amongst others). If the future some day will require you to move to another file format, you can do so then. In the meantime, for DNG to succeed the short-term benefits need to be bigger and clear. I don’t believe they are.”
Thom, I had a Kodak DC120 back in 1997. I converted most of the files from the proprietary format to jpg, but missed a few. Now I can’t find a single bit of software willing to convert or even read those old files.
By the way, I tried to use DNGconverter (and, for the records, digiKam) with a raw file from a Panasonic Lumix LX5 camera, but it fails.
Do anyone know if the problem is with my version? I have installed the Ubuntu 10.10 version, which is 1.4.0 (both digikam and kipi-plugins), with libkdcraw version
4.5.1.
Could an upgrade solve it? Any suggestion? Thanks!
Found by myself… LX5 is supported in 1.8.0. To use it in Ubuntu 10.10, you have to add the kde-backport ppa (see
https://launchpad.net/~kubuntu-ppa/+archive/backports).
It is a bit scary — it will ask you to do a partial upgrade — but for me all was ok.
Ah, log-out and log-in again after the upgrade, otherwise the digiKam albums come out empty (what a scare!)
“But there is no reason why you shouldn’t save your RAW files in the DNG format to be on the safe side.”
Safe side for what? If you don’t embed your original RAW you lose all ability to ever use the tools from your camera manufacturer or 3rd party tools that may struggle with DNG. DNG is a lossy format in that it only contains part of the meta information and that may or may not be the full story depending on bugs or limitations in the converter that existed at the time of the original conversion.
So DNG is only adding volume to your backup problem but is a solution to a non existing problem – as there now is an open source converter for almost any current RAW format to DNG there effectively is no need at all to convert to DNG *now* – because you can do that easily *when the need arises*. You can then be certain that you get the best possible DNG interpretation, something which you surely wouldn’t today (as the DNG specification is a work in progress).
DNG is just another reason for Adobe to put there name on something else. It’s all BS.
Apologies for the late response – I’ve just come across this post. I’m in search of a DNG workflow using Linux-only tools, if possible.
I’m aware of all the arguments for an against DNG, but am swung in favour mainly due to Peter Krogh’s arguments in his (excellent) book “The Dam Book”. He also expands on them as part of the team on http://www.dpbestflow.org (digital photography best practices and workflow).
The main argument for me is not about longevity, but about the ability to insert PIE (parametric image editing) information, plus multiple updated jpeg copies of the adjusted RAW file into the DNG, along with all the other associated metadata. This for me, means no sidecar files, no information that could get separated from the photo, and all metadata available in one location for me to be able to recreate the photo at a later date. You can also insert hashes into the DNG of the photos so that you can do immediate data verification of the photo portion of the file.
You still tie yourself to a non-destructive editor (take your pick) – but you retain all the information with the original file, and in a human-readable format since the PIE data and all is usually written and stored in xmp.
I realise that the only DNG converter available under Linux does not yet seem to do all these actions yet – but I hope that it will do soon.
You have to admit (although maybe I’m dead wrong about this) – there don’t seem to be many other file formats that keep all this data in one location, where I believe that it should be – in the file itself. I am certainly struggling to find one.
I know that DNG inserts an additional step into the workflow process, but it may well also simplify the longer-term effects of ensuring that all the data you need about your photo is stored WITH your photo. Sort of makes sense to me at least.
I am not connected to Adobe at all, but I am in desperate search of a long-term solution to my cataloguing needs where all my data is kept where I think it should be. Please post back if I have any major flaws in my thinking or if you feel that you could point me in the right direction on achieving this.
Thanks.
The ever changing DNG file (if you store that information inside it instead of the proper place which happens to be a sidecar file) will produce a backup nightmare and you will fill up your backup with redundant information by the bucketload (each little change in the image editing area will add the whole DNG file to the backup again and again although the effective change only is a few lines in the development recipe). Since DNG per se is a flawed concept (you are transcribing important information only partially, the rest is lost inside the DNG storage) you have to store and save the original RAW file anyway.
DNG happens to be a lousy format for archival and it’s a mediocre one at best for editing…
Karl,
Thanks for that. It’s an interesting argument – that it causes back-up problems for each change. I can see how that would happen.
I’m not yet convinced that “the proper place” for PIE data is in a sidecar file because I’m still concerned about losing or overwriting that (separated) data over time. Why do you believe that sidecar files are the answer, and how to you ensure that updated changes to the recipe in the sidecar file are maintained with the file over time? In addition, what about clashes where different PIE software tries to overwrite or use the same sidecar file (which I’ve read about online – a case of Bibble overwriting a Darktable sidecar, or was it the other way round?). In any case, there are many unresolved issues here that I would appreciate being clarified. You obviously have a good handle on this, so please feel free to expand further and help me learn.
I understand your comments about the data transcription flaws when moving RAW metadata to DNG. I’ve read a little about the DNG issue of keeping all metadata as a bit by bit data dump, regardless of the DNG converter’s ability to translate it effectively into useful metadata within the DNG file. I realise that this is far from perfect – but surely it’s just an issue to resolve, that should improve over time, right? As to having to save the original RAW file anyway, in terms of the meta-data, it’s all there inside the DNG file, it just needs to have software that can, a) read it (in the short term) and b) can translate it correctly in the first place (in the longer term). In the meantime, as a system that is not perfect, isn’t DNG still the best one that we have currently available?
In your opinion, is there anyone working on the ‘solution’ to the data-file problem itself? Or are people still working on playing catch-up in the FOSS world to get to something as workable as a Mac or PC solution before turning themselves to the underlying issue of which files to use over the longer term?
By data-file problem, I refer to the idea of finding a solution that allows for the following areas that need looking at:
1. Using RAW as an input
2. Allowing edits to be completed on the file and saving that ‘edit recipe’ where it can’t be lost (wherever that may be)
3. That allows multiple Parametric Image Editors to open, read and save additional information without clashes
4. Whilst also ensuring that all data is open for later reading on any software
5. Allows meta-data editing/updating/additions
6. Can be backed up effectively without losing any data, or as you say, without creating multiple copies of the same file that only have minor changes in PIE instructions?
7. Still allows proprietary information to be stored in it by the likes of Nikon or Canon (to allow them to differentiate themselves). I believe that this last one is very important – if we want a file format to actually be adopted widely, and preferably, upstream by the manufacturers themselves.
For my part, I believe that DNG covers most of these – and that nothing else I know of gets close (yet). Am I totally off the mark here? Are there other realistic competitors to the DNG format?
Your argument about editing and the problems of backup copies and their size is a very strong one. But for all that, I am not sure that your other arguments hold the same weight.
I do realise that all this is (relatively) new – and that this is very much in flux. Digital imaging is a young technology and all the problems are far from being resolved. Nevertheless, please help me understand/clarify the current status and its place on the road to image editing/cataloguing nirvana!
I look forward to hearing your comments.
Mark,
let me explain.
First of all DNG is a mess when it comes to meta data safety. It may copy some of the data but once it has been copied (well it’s interpreted to be precise) there is no way to recover from this process. Then there are discarded data areas (such as the masked areas of the sensor for which early DNG converters had no place to store it). Even if the latter isn’t an issue anymore (of which I am not certain about, most DNG converters are closed source), the lack of well defined process where a newer DNG converter can reach into an already existing DNG to dig out that propriarity data and then recreate the values once created by interpreting part of that data doesn’t exist.
Of your list DNG doesn’t solve quite a few of them – XMP meta data is read and rewritten by quite a few programs, some do discard XMP information they don’t know, no matter if this data is stored in an XMP sidecar or the DNG itself. If you only have the XMP sidecar file then there is a solution, which is to create generation copies of that file. Then you have at least a chance to recover from such a mishap. There even should be a way of having different sets of development recipies in parallel from which you could choose (by keeping several xmp sidecar files along the original RAW file) – for which there is no provision when embedding the XMP information in the DNG.
IMHO there isn’t a file format which allows for all of your points to be solved in the first place. I give DNG little chance to ever evolve into this all encompassing universal format – because it hasn’t done so yet, the big manufacturers didn’t embrace it. I can understand them in not doing so, because it limited their ability to innovate in the sensor area without spilling the beans to their competitors (as they would have had to extend the DNG standard well in advance before product release) – see the Tone Priority dynamic amplifier which, when used, IIRC will yield CR2 files which can’t be DNGized without loss of quality. The drawbacks of limiting yourself to DNG are thus becoming evident. I once embraced DNG much like you did until I discovered that for some of my pictures (I luckily kept the original files) it lost the proper black value data in the conversion, resulting in much worse conversions than the camera JPEG files, no matter how hard I tried and there wasn’t any way of ever recovering it except from the original file.
So as much as I like all the work that has been put in by the DigiKam people, the effort put in to create a converter to DNG is IMHO a complete and utter waste, opening a whole can of worms, especially playing into the hands of a would be monopolist in this area – as DNG development is, even open as it may seem, a propriarity format much like the original RAW files with all the drawbacks but no advantages.
With competent open source converters like RawTherapee around the chances that the ability to develop certain RAW file formats (which was the original argument for chosing DNG and which is listed errornously on the dpbestflow website) is never going to disappear, so the original RAW files have become the preferred archival format over the DNG. DNG only ever can be second best as the quality of the DNG always depends on the understanding of the original RAW file at the time of the creation of the DNG. Thus if DNG is to be used then only the DNG version which embeds the complete original RAW file is the way to proceed and IMHO they must reside on a read only media…
Karl,
Thanks for the response – good info to hear. It helps my understanding a lot. Of course, it doesn’t help resolve my issue of the large backlog of photos that I have sitting on my Linux system that need to be worked through…! And I’m holding you personally responsible for that! 😉
I still need to find a workable workflow on Linux that allows me to do what is needed, but keeps all the data available – and where I can access it, with as little possibility of associated file separation as possible…! D’oh. That’s another story though, and besides, if its waited this long, it can wait a little longer for me to resolve! Whilst solutions are available – I am reluctant to pursue one of them without a clear understanding of how I might migrate from one to another (more complete solution?) at a later date.
So then, I just want to clarify and state my understanding of what you say, and what I’ve gleaned from the internet so that we can agree on the status.
1. DNG is not 100% safe in terms of meta-data. Even if it copies meta-data that is doesn’t understand across to the DNG, there is nothing that currently successfully reads this “stored” data and could translate it back to the meta-data of the DNG at a later date. Closed-source converters may also not correctly save all metadata in the first place, although this may be a problem with closed-source software in general, and not necessarily DNG per se.
2. DNG converts some data from the RAW on creation (hot pixels, I believe, as a minimum), and as such, is not a fully secure system in terms of maintaining all original data or being able to recreate your original RAW data from the DNG. This conversion provides potential for screwing up your image file – as you found out.
3. Currently, all RAW formats have been reverse-engineered for visibility within the open-source world – and these could now (in theory) always remain available to view and manipulate because the source code is available now. However, for new RAW file formats, this will continue to rely on the ability of open-source teams to reverse engineer them.
4. DNG is unlikely to evolve considerably from its current position, and its slow pace of change and its limited adoption within the industry mean that it may not change successfully to allow future adoption of necessary changes. As a challenge here, though, even if we had an open-source alternative file format, wouldn’t it suffer the same issues?
5. There is no file format that currently addresses all of these issues (or those that I mentioned above) and certainly none that also address non-destructive file editing and saving, plus providing good backup systems and allowing for future growth and/or change in the industry.
Would this be about the state of the current situation as it stands now, at the end of 2011?
You make a valid point that if a RAW format can be read using Dave Coffin’s excellent dcraw (which I believe is the basis of much, if not all, open source RAW management) then there is no need for an intermediary system such as DNG. This, of course, is true only as long as dcraw is maintained and continues to do an excellent job of reverse-engineering new RAW formats. For existing formats it will be fine, but for new formats, it will always require that reverse-engineering step unless manufacturers open up a little more about the image data side of their RAW formats. Is this a correct interpretation of the situation?
It would appear that all of this discussion only points out (or highlights more thoroughly) the inadequacy of the current systems to stand up to creating and maintaining a future-proof file system that works hand-in-hand with an expandable digital workflow. To explain ‘expandable’, I mean – a file-format or workflow system that hides most of the difficulties for general consumers, but allows padding of the workflow (for prosumers and professionals) that includes adequate backup strategies, non-destructive editing, export and then destructive editing (if needed), plus whatever a more dedicated professional requires as part of their workflow.
I can see from both what you say, and my own research, that whilst DNG has laudable goals, the implementation still leaves something to be desired – and that leaves aside any of other people’s concerns about Adobe’s allegedly monopolistic intentions etc.
So, back to the question I asked earlier – is anyone working on the answer for the future?
Is anyone trying to solve this actual issue, or are they too busy working on their open source software dealing with today’s issues, and not having the time to think about the future? This is not a criticism of those fine people working very hard to give us excellent open-source software for photo manipulation and management. They are superstars, all of ’em! And let’s never forget it. But when you’re down “deep and dirty” amongst the code and working on getting software to display a file format correctly or manipulate the image, it might not always be easy to step back and see the bigger picture about whether this is the right file format in the first place!
In your opinion, what is the best direction for the ‘industry’ to be headed in terms of a suitable file format or wrapper that allows for the current and future requirements of a digital workflow?
The dpbestflow website (and Krogh) call DNG a “digital job wrapper”. From what we discuss here, it would appear that it does more than that, such that it does not just ‘wrap’ a RAW file, it ‘converts’ it too.
What if we were to look at the idea of a different system? What if we were to create a new kind of “digital image wrapper”?
And although DNG may not be a perfect solution, I still kind of like some of the ideas that originally brought it to the fore… For example, the idea of having a single system or level playing ground that acts as a starting point for all later work on a digital image file. A stable base to make any edits from (non-destructive or otherwise), where all future software could start from. And another point: to provide a single system of data management and archiving that can be used for back-up that can then be opened at all points in the future etc.
To point to something that you alluded to earlier – although a file format such as DNG is an intermediate step,there is no reason why a ‘good’ intermediate step might not bring advantages. How can the community make a stable basis for future work, whilst gaining advantages of this platform and not gain the drawbacks? Is it possible to draw this together and create a suitable alternative to DNG that fits this bill?
Has anyone tried this, or has this opportunity been missed so far?
What are your thoughts about finding a suitable stable platform as a basis for all future image software to work from? Would it allow simplification for PIE-ware and meta-data handling for catalogue software? Would it also allow better longevity for images?
At the very least, I’m not yet convinced of any arguments other than the thought that current systems are still an immature solution.
What would a perfect(?!) solution to this problem look like?
Is this an opportunity for the Open Source community to stop following and start leading in this industry? 😛
I’m enjoying this discussion and debate – please do keep pushing back on me…!
I’m using this DNG convertor now, but somehow a few NEF images are skipped in the process. When I select them once more and run DNG again they are converted ok. With lots of images I had to do this twice to get all images converted. Must be a bug imho.
Hi, I have just been trying the DNG converter on Fujifilm x100s RAF files which unfortunately appear not to be supported. Any hope of including them in the next release of digikam? Please.