Autarchy of the Private Cave

Tiny bits of bioinformatics, [web-]programming etc

  • Related entries

    No related content found.

    IOMMU: This costs you 64 MB of RAM

    30th September 2009

    If you have happened to observe similar messages in your dmesg:

    aperture

    [ 0.004000] Checking aperture…
    [ 0.004000] No AGP bridge found
    [ 0.004000] Node 0: aperture @ 20000000 size 32 MB
    [ 0.004000] Aperture pointing to e820 RAM. Ignoring.
    [ 0.004000] Your BIOS doesn’t leave a aperture memory hole
    [ 0.004000] Please enable the IOMMU option in the BIOS setup
    [ 0.004000] This costs you 64 MB of RAM
    [ 0.004000] Mapping aperture over 65536 KB of RAM @ 20000000

    and you are using AMD-based system w/o AGP video, then my advice is: just leave that as is, do not bother “improving”! Any tinkering with kernel boot options won’t do you any good, as the kernel has already done the best it could.

    Just a note: all those messages at the top of the post should only happen if you have 4 or more GiBs of RAM. If you have less than that, and do have those messages – my experience might be inappropriate for your case.

    Another note: my BIOS does not have any IOMMU settings (or “Memory hole remapping” settings), so I didn’t try that. You should check if your BIOS has IOMMU-related options first, just as kernel message suggests.

    Read on for details.

    My system’s kernel at the moment of writing is 2.6.30-1-amd64 (Debian testing), running on MSI’s 790GX-G65 with integrated Radeon HD 3300 video, with 4 GiB DDR3 RAM and a quad-core Phenom II CPU.

    Following are the options you shouldn’t try on a similar system:

    • passing iommu=noagp kernel boot option. If you run dmesg|grep -i agp, most likely you will see something in the lines of

      $ dmesg|grep -i agp
      [ 0.004000] No AGP bridge found
      [ 0.392098] PCI-DMA: Disabling AGP.
      [ 0.392142] PCI-DMA: Reserving 64MB of IOMMU area in the AGP aperture
      [ 0.774847] Linux agpgart interface v0.103

      which tells you that your kernel has automatically discovered the absence of AGP bridge, disabled it and is using 64MB of AGP aperture for IOMMU. On my system, adding that boot option had no effect.

    • passing iommu=off kernel boot option. When I did this, there were loads of “nommu overflow” errors during boot, so I didn’t wait until GDM start and rebooted immediately. Clearly, tons of “overflow” errors won’t do any better than a harmless “This costs you 64 MB of RAM”. Also, systems with over 3 GiB RAM do need IOMMU.
    • passing iommu=noaperture kernel boot option. On my system, this option has taken 1 more MiB from my available RAM (as reported by free -m or htop), and also enabled software double-buffering IOMMU method (in dmesg, a string similar to SW TIOLB). Performance-wise this is worse than AMD’s GART-based IOMMU, and you loose 1 more MiB of RAM ;), so don’t do this.
    • passing iommu=noagp,noaperture for me had the same effect as passing iommu=noaperture alone, for the reason from point #1 of this list.
    • I haven’t tried any of the memaper=[1|2|3|...] options, as I had no problems with the default 64 MiB.
    • passing mem=4G may remove that message (I do not recollect if it did for me), but will eat a good deal more of your available RAM (might be because devices are now mapped within those 4G, and not above). So don’t do this.

    I’m not aware of any other solutions to the missing 64 MiB RAM message. There were notices of using pci=nommconf boot option, but that is too much for such a simple message – unless you have problems booting.

    Below I’m pasting IOMMU-relevant part of the linux-source-2.6.30/Documentation/x86/x86_64/boot-options.txt file, which I believe is the source of ultimate knowledge :) You should read that file whenever you feel like tinkering with boot options of your AMD64 machine.

    I ***highlighted*** some fragments, to support the points from the list above.

    IOMMU (input/output memory management unit)
     Currently four x86-64 PCI-DMA mapping implementations exist:
       1. : use ***no hardware/software IOMMU at all***
          (e.g. ***because you have < 3 GB memory***).
          Kernel boot message: "PCI-DMA: Disabling IOMMU"
       2. : ***AMD GART based hardware IOMMU***.
          Kernel boot message: "PCI-DMA: using GART IOMMU"
       3.  : Software IOMMU implementation. Used
          e.g. ***if there is no hardware IOMMU*** in the system and it is need because
          you have ***>3GB memory*** or told the kernel to us it (iommu=soft))
          Kernel boot message: "PCI-DMA: Using software bounce buffering
          for IO (SWIOTLB)"
       4.  : IBM Calgary hardware IOMMU. Used in IBM
          ***pSeries and xSeries servers***. This hardware IOMMU supports DMA address
          mapping with memory protection, etc.
          Kernel boot message: "PCI-DMA: Using Calgary IOMMU"
     iommu=[][,noagp][,off][,force][,noforce][,leak[=]
    	[,memaper[=]][,merge][,forcesac][,fullflush][,nomerge]
    	[,noaperture][,calgary]
      General iommu options:
        off                Don't initialize and use any kind of IOMMU.
        noforce            Don't force hardware IOMMU usage when it is not needed.
                           (default).
        force              Force the use of the hardware IOMMU even when it is
                           not actually needed (e.g. because < 3 GB memory).
        soft               Use software bounce buffering (SWIOTLB) (default for
                           Intel machines). This can be used to prevent the usage
                           of an available hardware IOMMU.
      iommu options only relevant to the AMD GART hardware IOMMU:
                     Set the size of the remapping area in bytes.
        allowed            Overwrite iommu off workarounds for specific chipsets.
        fullflush          Flush IOMMU on each allocation (default).
        nofullflush        Don't use IOMMU fullflush.
        leak               Turn on simple iommu leak tracing (only when
                           CONFIG_IOMMU_LEAK is on). Default number of leak pages
                           is 20.
        memaper[=]  Allocate an own aperture over RAM with size 32MB<4GB.
                           DAC is used with 32-bit PCI to push a 64-bit address in
                           two cycles. When off all DMA over >4GB is forced through
                           an IOMMU or software bounce buffering.
        nodac              Forbid DAC mode, i.e. DMA >4GB.
        panic              Always panic when IOMMU overflows.
        calgary            Use the Calgary IOMMU if it is available
      iommu options only relevant to the software bounce buffering (SWIOTLB) IOMMU
      implementation:
        swiotlb=[,force]
                    Prereserve that many 128K pages for the software IO
                           bounce buffering.
        force              Force all IO through the software TLB.
      Settings for the IBM Calgary hardware IOMMU currently found in IBM
      pSeries and xSeries machines:
        calgary=[64k,128k,256k,512k,1M,2M,4M,8M]
        calgary=[translate_empty_slots]
        calgary=[disable=]
        panic              Always panic when IOMMU overflows
        64k,...,8M - Set the size of each PCI slot's translation table
        when using the Calgary IOMMU. This is the size of the translation
        table itself in main memory. The smallest table, 64k, covers an IO
        space of 32MB; the largest, 8MB table, can cover an IO space of
        4GB. Normally the kernel will make the right choice by itself.
        translate_empty_slots - Enable translation even on slots that have
        no devices attached to them, in case a device will be hotplugged
        in the future.
        disable= - Disable translation on a given PHB. For
        example, the built-in graphics adapter resides on the first bridge
        (PCI bus number 0); if translation (isolation) is enabled on this
        bridge, X servers that access the hardware directly from user
        space might stop working. Use this option if you have devices that
        are accessed from userspace directly on some PCI host bridge.
    Share

    Leave a Reply

    XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>