Most of the people of Westeros are sea-faring, most notable among them are the Ironborn and the Reachmen.

The Westerosi often trade with the Free Cities and even with the Slaver cities of the East. The Ironborn also raid in the far east.

How do they find their bearings to navigate their ships correctly to foreign and domestic ports? Do they sail along the coastline to navigate with the help of the land marks (Although that wouldn't work when sailing between continents) or do they navigate using the stars?

They use the stars

Maester Luwin sighed. "I can teach you history, healing, herblore. I can teach you the speech of ravens, and how to build a castle, and the way a sailor steers his ship by the stars. I can teach you to measure the days and mark the seasons, and at the Citadel in Oldtown they can teach you a thousand things more. But, Bran, no man can teach you magic."

_{A Game of Thrones, Bran VI}

Davos is also very familiar with the stars and refers to them as smuggler's stars suggesting he used them for navigation back in his smuggling days. He also refers to some of the constellations by name.

A half moon was sliding in and out amongst thin high clouds, and Davos could see familiar stars. There was the Galley, sailing west; there the Crone's Lantern, four bright stars that enclosed a golden haze. The clouds hid most of the Ice Dragon, all but the bright blue eye that marked due north. The sky is full of smugglers' stars. They were old friends, those stars; Davos hoped that meant good luck.

_{A Storm of Swords, Davos VI}

And landmarks/coastlines

Davos tells of how he knows the shapes of the coastlines, most likely having had sailed around them enough times and used them for navigation purposes.

"There's much I don't understand," Davos admitted. "I have never pretended elsewise. I know the seas and rivers, the shapes of the coasts, where the rocks and shoals lie. I know hidden coves where a boat can land unseen. And I know that a king protects his people, or he is no king at all."

_{A Storm of Swords, Davos VI}

Tyrion commands the captain who is taking Myrcella to follow the coastline, though it's not clear if this is for protection or navigation.

He cleared his throat. "You know your orders, Captain."

"I do, my lord. We are to follow the coast, staying always in sight of land, until we reach Crackclaw Point. From there we are to strike out across the narrow sea for Braavos. On no account are we to sail within sight of Dragonstone."

_{A Clash of Kings, Tyrion IX}

The following quote tells of a priest telling Victarion to use the coastline to navigate to find some ships to pillage.

"The captain commands, and I obey," said Moqorro. The crew had taken to calling him the Black Flame, a name fastened on him by Steffar Stammerer, who could not say "Moqorro." By any name, the priest had powers. "The coastline here runs west to east," he told Victarion. "Where it turns north, you will come on two more hares. Swift ones, with many legs."

_{A Dance with Dragons , Victarion I}

Sailors also use Myrish lenses/far eyes which is essentially a telescope. These would have been used for various reasons such as scouting the enemy but they are also used for spotting land. As can be seen when Catelyn sails for King's LKanding.

High overhead, the far-eyes sang out from the rigging. Captain Moreo came scrambling across the deck, giving orders, and all around them the Storm Dancer burst into frenetic activity as King's Landing slid into view atop its three high hills.

_{A Game of Thrones, Catelyn IV}

Note in the above quote "far-eyes" is referring to the person who is using the far eyes and sitting in the crows nest. So, just to be complete here is a reference to a captain owning one.

This time it was a Myrish cog named Dove, on her way to Yunkai by way of New Ghis with a cargo of carpets, sweet green wines, and Myrish lace. Her captain owned a Myrish eye that made far-off things look close—two glass lenses in a series of brass tubes, cunningly wrought so that each section slid into the next, until the eye was no longer than a dirk. Victarion claimed that treasure for himself.

_{A Dance with Dragons, Victarion I}