OSDev.org

The Place to Start for Operating System Developers
It is currently Fri May 17, 2024 5:04 am

All times are UTC - 6 hours




Post new topic Reply to topic  [ 29 posts ]  Go to page Previous  1, 2
Author Message
 Post subject:
PostPosted: Thu May 24, 2007 10:56 am 
Offline
Member
Member
User avatar

Joined: Mon Dec 11, 2006 10:54 am
Posts: 190
Location: Dunedin, New Zealand
Just had an idea.
If I made each page, 2bits I could decrease the checking time significantly.

00 = free.
01 = unusable, due to smaller block.
10 = unusable, part of a larger block.
11 = used.

Still would need to do checking though, lots of it
Id still only have one bitmap though!

~zeii


Top
 Profile  
 
 Post subject:
PostPosted: Thu May 24, 2007 11:28 am 
Offline
Member
Member

Joined: Mon Apr 09, 2007 12:10 pm
Posts: 775
Location: London, UK
How would you go about using the unused tags? I can't quite see how they'd work.

I've attached a file containing 6 images of how a buddy system would work with the following commands. The images might look familiar! Note that red means the area is free, blue is used.

Code:
void *a = palloc(64kB);
void *b = palloc(4kB);
free(a);
void *c = palloc(4kB);
a = palloc(64kB);


Note that at the start, only the largest block is marked free. This is split as needed. When the second palloc(4kB) is made (void *c) then there is a free block of 4kB (right next to b) which is used for it, even though the first 64kB is actually free. If you just use a bitmap, you'd just go looking for a free 4kB block, note that the first one is free, return that and think no further.

Regards,
John.


Attachments:
buddy diag1.GIF
buddy diag1.GIF [ 29.33 KiB | Viewed 1959 times ]
Top
 Profile  
 
 Post subject:
PostPosted: Thu May 24, 2007 11:33 am 
Offline
Member
Member
User avatar

Joined: Mon Dec 11, 2006 10:54 am
Posts: 190
Location: Dunedin, New Zealand
Actually, No, im being a retard :P.
Lack of caffiene and/or sleep.
Since they still share the same bitmap, those codes are meaningless.
Even if you do crazy math and bittwiddling.

So, let me get this straight ;).
You have bitmaps for sizes 4K through 4M.

If you have a 8MB System you will need:

4K Map: 256b.
8k map: 128b
16k map: 64b
32k map: 32b
64k map: 16b
128k map: 8b
256k map: 4b
512k map: 2b
1M Map: 1b
2M Map: 4 bits
4M Map: 2 bits

Riiight?
~zeii
PS: Hehehe, Familiar indeed! :)


Top
 Profile  
 
 Post subject:
PostPosted: Thu May 24, 2007 11:44 am 
Offline
Member
Member

Joined: Mon Apr 09, 2007 12:10 pm
Posts: 775
Location: London, UK
zeii wrote:
So, let me get this straight ;).
You have bitmaps for sizes 4K through 4M.


Yes, although my allocate_buddy(addr, smallest_page, largest_page) allows me to create any number of buddies from smallest_page to largest_page. I have just picked 4kB and 4MB as they are typical page sizes.

zeii wrote:
If you have a 8MB System you will need:

4K Map: 256b.
8k map: 128b
16k map: 64b
32k map: 32b
64k map: 16b
128k map: 8b
256k map: 4b
512k map: 2b
1M Map: 1b
2M Map: 4 bits
4M Map: 2 bits


Actually my buddies are padded with zeros to the next dword boundary, so the next one starts on a dword boundary. Therefore in this example everything from 256kB through 4MB will occupy 4 bytes. Otherwise, its essentially correct. You can see that you only need 510 bytes to store the buddies, or to put it another way, a bootsector!

[wiki]Page_Frame_Allocation[/wiki] probably has a better description of all the different techniques available.

Regards,
John.


Top
 Profile  
 
 Post subject:
PostPosted: Thu May 24, 2007 11:49 am 
Offline
Member
Member
User avatar

Joined: Mon Dec 11, 2006 10:54 am
Posts: 190
Location: Dunedin, New Zealand
So your saying... every bit, in my system is 4bytes in your system?
Or am I misunderstanding what you mean by dword aligned?

~z


Top
 Profile  
 
 Post subject:
PostPosted: Thu May 24, 2007 11:56 am 
Offline
Member
Member

Joined: Mon Apr 09, 2007 12:10 pm
Posts: 775
Location: London, UK
The start of each buddy bitmap is dword aligned.

So say the 256kB bitmap is at 0x1000, it runs for 4 bytes
The 512kB bitmap is at 0x1004, it runs for 2 byes, but I zero pad it to 4 so that
The 1MB bitmap is at 0x1008, for 1 byte, again zero padded to 4 bytes
The 2MB bitmap is at 0x100C for 4 bits but zero padded to 4 bytes
The 4MB bitmap is at 0x1010 for 2 bits but zero padded to 4 bytes.

It just means that I can check 4 bytes at a time. If the 512kB bitmap wasn't padded, then the 1MB one would start at 0x100A. Testing the dword at 0x1008 would test the end of the 512kB bitmap and the start of the 1MB bitmap, which isn't a good thing to do in the same test.

Regards,
John.


Top
 Profile  
 
 Post subject:
PostPosted: Thu May 24, 2007 12:00 pm 
Offline
Member
Member
User avatar

Joined: Mon Dec 11, 2006 10:54 am
Posts: 190
Location: Dunedin, New Zealand
Fair enough :)
Thanks for this duel! :) Its given me a good bunch of ideas.
(Ysee, im obsessed with making things as tiny as possible!)

:) I shall ponder (And drink copious amounts of caffiene, ah... good for the soul)

~z


Top
 Profile  
 
 Post subject:
PostPosted: Thu May 24, 2007 12:11 pm 
Offline
Member
Member
User avatar

Joined: Mon Dec 11, 2006 10:54 am
Posts: 190
Location: Dunedin, New Zealand
Important thing is, though now.. Is how do manage Virtual memory - if we use Buddy for Physical?

Personally, Im not so worried about Physical memory becoming fragmented, Im more concerned about Virtual memory becoming fragmented.

If Physical gets all fragged, we can alias it in such a way that to the Virtual space, it seems completely contiguous. Sadly, we cant do the same for Virtual memory - if its fragmented, it is fragmented.

So, it comes to be ... Why not use Buddy allocation to track the regions in a Virtual address space? At this point of the game, you could use a tree structure (with all of its extra overhead) or you could stick with the bitmaps.

I am curious to how people solve this problem. I would much rather deal with Internal fragmentation than External.

Pros / Cons for Buddy allocator tracking virtual addresses?
(That being said, that Virtual addresses are all private to their owning context, except for the Kernel - which has its own weird context, that exists in every other).
~zeii


Top
 Profile  
 
 Post subject: A partial example of a buddy allocator.
PostPosted: Thu May 24, 2007 5:09 pm 
Offline
Member
Member
User avatar

Joined: Tue Nov 09, 2004 12:00 am
Posts: 843
Location: United States
The buddy allocator from what I understand which could be wrong will work something like below (being a partial example):

Code:
//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#define MAXSTACKITEMS ((4096-2-4-4)/4)
struct sStack{
   uintptr_t    addr[MAXSTACKITEMS];
   uint16_t   index;
   struct sStack   *next, *prev;
};
struct sGroup{
   uint32_t   freeCount;
   struct sStack *free;
   struct sStack *used;
};
//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#define REDZONE_CLUSTERCOUNT 5
#define YELLOWZONE_CLUSTERCOUNT 10
#define GREENZONE_CLUSTERCOUNT 20
#define MAXPAGECLUSTERSIZE 7
static struct sGroup group[MAXPAGECLUSTERSIZE];
//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/// The functions (private and public).
static void fill(uint8_t groupIndex);
static void push(uint8_t groupIndex, uintptr_t address);
static uintptr_t pop(uint8_t groupIndex);
void *valloc(uint32_t pageCount);
//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/// Brings a group that is below the GREENZONE in cluster count into the GREENZONE.
static void fill(uint8_t groupIndex)
{
   uint32_t needed;
   /// take a cluster from a higher level and fill this level with it.
   if((groupIndex+1) > MAXPAGECLUSTERSIZE){
      /// we are out of memory?
      PROBLEM;
      for(;;);
   }
   if(group[groupIndex].freeCount > GREENZONE_CLUSTERCOUNT){
      // ignore it?
      return;
   }
   for(needed = (GREENZONE_CLUSTERCOUNT - group[groupIndex].freeCount) / 2; needed > 0; --needed){
                /// this is the splitting for the lower half of the cluster.
      push(groupIndex, pop(groupIndex+1));
                /// this is the splitting for the upper half of the cluster.
                push(groupIndex, pop(groupIndex+1) + (groupIndex * 4096));
   }
   return;
}
//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/// Pushes a cluster specified at 'address' into the group specified by 'groupIndex'.
static void push(uint8_t groupIndex, uintptr_t address)
{
   struct sStack *nstack;
   if(group[groupIndex].free->index == MAXSTACKITEMS){
      // create a new array for the list.
      nstack = (struct sStack*)pop(1);
      nstack->prev = group[groupIndex].free;
      group[groupIndex].free->next = nstack;
      nstack->next = 0;
      nstack->index = 0;
      group[groupIndex].free = nstack;
   }
   ++group[groupIndex].freeCount;
   group[groupIndex].free->addr[(group[groupIndex].free->index)++] = address;
   return;
}
//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/// Pops a cluster from the group specified with 'groupIndex'.
static uintptr_t pop(uint8_t groupIndex)
{
   struct sStack *tmp;
   if(group[groupIndex].free->index == 0){
      tmp = group[groupIndex].free;
      group[groupIndex].free = group[groupIndex].free->prev;
      group[groupIndex].free->next = 0;
      push(1, (uintptr_t)tmp);
      if(group[groupIndex].freeCount < REDZONE_MINCLUSTERCOUNT){
         /// schedule a replenishment or immediately replenish.
         fill(groupIndex);
      }
      // if(grou[groupIndex].freeCount < YELLOWZONE_MINCLUSTERCOUNT){
                // .... something here.. maybe a scheduler and not right now.
   }
   --group[groupIndex].freeCount;
   return group[pageCount].free->addr[--(group[groupIndex].free->index)];
}
//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/// Almost a plain shadow of pop(). Missing error handling and such other things.
void *valloc(uint32_t pageCount)
{
   return (void*)pop(pageCount);
}



I use a stack to store free clusters of pages. I then made the stack have the ability to hold more than MAXSTACKITEMS by placing a next and prev pointer. So a stack is really a cluster of arrays holding MAXSTACKITEMS each. Not sure but I figured it would be a safe bet since I have never implemented a buddy allocator type system before.

Then pop and push functions handle deleting unused arrays in each stack array cluster and making new arrays in each cluster. They also check if the stack is going below the YELLOW and RED threshold counts. If they do then the fill function is called which computes the number of clusters needed from the page cluster group above the current page cluster group. Then the pop and push is used to convert the pages from one group to another.

You might want to only scheduler a kernel thread to handle filling some page clusters into group in the YELLOW range and do a immediate fill when in the RED range.

I also have some syntax highlighting here for the code (above).
http://kmcguire.jouleos.galekus.com/dok ... or_example

_________________
Projects


Top
 Profile  
 
 Post subject:
PostPosted: Fri May 25, 2007 8:46 am 
Offline
Member
Member

Joined: Mon Apr 09, 2007 12:10 pm
Posts: 775
Location: London, UK
zeii wrote:
Thanks for this duel! Smile Its given me a good bunch of ideas.


My pleasure!

zeii wrote:
Why not use Buddy allocation to track the regions in a Virtual address space? At this point of the game, you could use a tree structure (with all of its extra overhead) or you could stick with the bitmaps.


That depends on what you mean by regions for a virtual address space.

Do you mean regions e.g. text, data, bss, stack and heap or do you mean supporting a malloc-like interface within the heap. I can only comment on a bitmap based system, but if you're talking about a malloc interface, remember it is possible to call malloc(1) asking for only one byte. This would mean you need to represent every byte with a bit in the lowest buddy, or in other words use up almost a quarter of the total heap size with buddies.

Its not so bad if you mean use it for page-sized regions like text, data etc. In that case, its still a lot however. Consider we let processes use 2GB of address space. That corresponds to 128kB of buddies per process, regardless of the amount of physical ram in the machine.

So after arguing that size isn't important for buddies to represent physical memory, I'm now arguing the exact opposite for virtual memory! :wink: This is because the total amount of virtual memory in a machine is 4GB * no. of processes (or far larger for long mode), whereas the physical memory where you actually store the structures is much less.

So the only real con I can think of is size. Actually, the size would be more, because you would probably want to use more than one bit to represent a page when dealing with virtual memory. You have to think of access rights etc.

The pros are reducing fragmentation and speed. However, I think these can be accomplished by using currently recognised systems for chopping up virtual memory. I use a static array of region structures for text, data, bss, stack and heap, which define start and end points as well as any flags for that region. The heap section can be increased by the user call brk(), which my user-space (i.e. resides in my libc) malloc() calls (actually I use dlmalloc). The stack region can automatically extend downwards if needed.

Then, my page-fault function merely checks this list to see if a certain region is allocated to the process, and if so maps a page (handily provided by the physical memory allocator).

Regards,
John.


Top
 Profile  
 
 Post subject:
PostPosted: Fri May 25, 2007 6:43 pm 
Offline
Member
Member
User avatar

Joined: Sat Jan 15, 2005 12:00 am
Posts: 8561
Location: At his keyboard!
Hi,

jnc100 wrote:
Do you mean regions e.g. text, data, bss, stack and heap or do you mean supporting a malloc-like interface within the heap. I can only comment on a bitmap based system, but if you're talking about a malloc interface, remember it is possible to call malloc(1) asking for only one byte. This would mean you need to represent every byte with a bit in the lowest buddy, or in other words use up almost a quarter of the total heap size with buddies.


It's common for malloc to allocate 8 bytes when asked to allocate less (and also common for malloc to "round up" to the nearest 8 byte boundary), for alignment purposes.

jnc100 wrote:
So after arguing that size isn't important for buddies to represent physical memory, I'm now arguing the exact opposite for virtual memory! :wink: This is because the total amount of virtual memory in a machine is 4GB * no. of processes (or far larger for long mode), whereas the physical memory where you actually store the structures is much less.


Not quite - the amount of space being managed by malloc depends on the process, and it doesn't manage the entire 4 GB (or 3 GB?). For example, malloc might start managing 16 MB of space, and get more space/RAM if it needs to, and free space/RAM when it can.


For physical memory management, the first thing you'd need to decide is if you care about where pages are allocated. Most physical memory allocations are 4 KB and it doesn't matter where the page comes from. Free page stack/s are the best option for this case - they're extremely fast and only costs 4 bytes (plus 4 bytes per free page, that's stored in otherwise free RAM and therefore doesn't matter).

Buddy allocators, bitmaps, etc should only be used if the location of allocated pages matters. Typically this is only for DMA buffers, which represent a small number of allocations and a small amount of RAM. IMHO it makes sense to split physical memory into several zones, and use free pages stack/s in one zone to make the common case fast and efficient, and use a buddy allocator or something in another zone for when the location of allocated pages matter. For example, I normally use a bitmap for memory below 16 MB, and free page stacks for anything above 16 MB.

The only other consideration for physical memory is CPU caches, where a physical page needs to go in a certain area of the cache and it's better to be using different areas of the cache instead of the same area of the cache. To get around this some OSs use page colouring (e.g. one free page stack per page colour).

AFAIK Linux doesn't support page colouring, so contiguous linear pages might be using physical pages that use the same part of the cache, which reduces cache efficiency. To get around this they (mis)use a buddy allocator to reduce the problem (so that contiguous linear pages are more likely to be contiguous physical pages, which use differnt parts of the cache). Also, Linux wastes most of the first 16 MB of RAM by storing a large (4 MB or more) kernel in it, so having a specific area of physical memory for allocations that need contiguous physical pages doesn't work so well. IMHO these poor design descisions makes a buddy allocators one of the best options for Linux, but without the poor design descisions a buddy allocator may not be the best option... ;)


Cheers,

Brendan

_________________
For all things; perfection is, and will always remain, impossible to achieve in practice. However; by striving for perfection we create things that are as perfect as practically possible. Let the pursuit of perfection be our guide.


Top
 Profile  
 
 Post subject:
PostPosted: Sun May 27, 2007 4:32 am 
Offline
Member
Member
User avatar

Joined: Mon Dec 11, 2006 10:54 am
Posts: 190
Location: Dunedin, New Zealand
Im particularly worried about KERNEL space. :P

In Zenobit, the Kernel is completely independant from a fixed Context. It exists in a little region simply named Kernel space.

This Kernel space, has a finite amount of Virtual pages that can be used. A standardish MALLOC-type system is NEVER used here, instead - it uses a Pool system for Memory Management.

So... While I could use REGION type stuff for the actual Userspace stuff, I cannot use it for the Kernel space stuff, there is far too much stuff that has been designed to work with just being given some space and left to its own devices.

So, I was thinking - Why not have Buddy allocation to track Physical memory (Since, its fast and can deal with logical sizes pretty well (8/16k anyone?). Have the Kernel space governed by a Buddy allocator itself, too. Everything else could be done with the standard regiony type crap.

That way, Kernelspace will not get fragmented. Since it relies heavily on Pool allocation systems, Internal allocation is kinda.. not a gigantic deal for me.

~z


Top
 Profile  
 
 Post subject:
PostPosted: Sun May 27, 2007 9:16 am 
Offline
Member
Member
User avatar

Joined: Sat Jan 15, 2005 12:00 am
Posts: 8561
Location: At his keyboard!
Hi,

zeii wrote:
So, I was thinking - Why not have Buddy allocation to track Physical memory (Since, its fast and can deal with logical sizes pretty well (8/16k anyone?).


Buddy allocators are slower than free pages stacks, they waste more useful RAM, and they're more complex to implement. The only advantage they have is that a buddy allocator would handle allocations that need contiguous physical pages, but most of the time you don't need contiguous physical pages anyway.

zeii wrote:
Have the Kernel space governed by a Buddy allocator itself, too. Everything else could be done with the standard regiony type crap.

That way, Kernelspace will not get fragmented. Since it relies heavily on Pool allocation systems, Internal allocation is kinda.. not a gigantic deal for me.


Does it matter if kernel space uses fragmented physical pages (if linear space isn't fragmented)? Typically the answer here would be "no, I'm using paging and for most things it doesn't make any difference if physical memory is extremely fragmented".

So, why would you want to use something slower like a buddy allocator for most things, when something faster and simpler (e.g. free page stack/s) can be used for most things instead?

I can fully understand using something like a buddy allocator for the rare occasions when you must have contiguous physical pages (and therefore must use something slower), but apart from that it makes no sense at all IMHO.

For a computer with 2 GB of RAM, about 8% of physical memory could be managed using a slower buddy allocator, and 92% of phyical memory could be managed by faster free page stack/s. If the free page stacks are (on average) 5 times faster than the buddy allocator, then this would make your OS's physical memory management about 4.5 times faster overall.


Cheers,

Brendan

_________________
For all things; perfection is, and will always remain, impossible to achieve in practice. However; by striving for perfection we create things that are as perfect as practically possible. Let the pursuit of perfection be our guide.


Top
 Profile  
 
 Post subject:
PostPosted: Sun May 27, 2007 10:04 am 
Offline
Member
Member
User avatar

Joined: Mon Dec 11, 2006 10:54 am
Posts: 190
Location: Dunedin, New Zealand
I agree entirely, buddy allocation used when you DONT need contiguous stuff is POINTLESS.

;) Which is the point.

Paging can only do so much. Zenobit currently uses a bitmap-stack (on average, it is constant speed for allocation, until ram is exhausted...). Im deciding to move Physical allocations to a nicer Page-stack, like the ones you speak of.

Problem is, Kernel space is Virtual - and it only has so many pages that it can use. Its already in virtual space, so I cant do that much - except use something that can ensure as little external-fragmentation as possible, which is why I intend to use a buddy allocator for tracking kernel space.

~z


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 29 posts ]  Go to page Previous  1, 2

All times are UTC - 6 hours


Who is online

Users browsing this forum: Bing [Bot] and 16 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group